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ABSTRACT 


Systems  engineering  (SE)  competencies  are  defined  based  on  the  knowledge,  skills,  and 
abilities  (KSAs)  necessary  for  a  systems  engineer  to  perform  tasks  related  to  the 
discipline.  Proficient  systems  engineers  are  expected  to  be  able  to  integrate,  apply,  and  be 
assessed  on  these  KSAs  as  they  develop  competencies  through  their  education  and 
training,  professional  development,  and  on-the-job  experience.  The  research  conducted 
by  the  Naval  Postgraduate  School,  assessed  where  SE  graduate  students  stood  as  far  as 
developing  the  necessary  competency  levels  they  need  to  be  successful  systems 
engineers.  A  survey  methodology  was  used  to  achieve  this  objective.  Systems 
engineering  students  enrolled  in  SE  courses  at  the  Naval  Postgraduate  School  represented 
the  population  surveyed. 

Survey  items  were  written  with  the  intent  to  capture  self-efficacy  for  knowledge 
and  skill  sets  as  a  subset  of  the  overall  set  of  competencies  required  for  systems 
engineering,  namely  critical  thinking,  systems  engineering,  teamwork  and  project 
management.  A  total  of  four  surveys  were  administered  to  two  SE  cohorts.  Results  show 
that  self-efficacy  in  systems  engineering  can  be  reasonably  assumed  to  be  positively 
affected  by  a  graduate  level  educational  program.  The  implications  of  the  research  can  be 
used  to  develop  structured  curriculum  content,  assessment,  and  continuous  process 
improvement  techniques  related  to  the  development  of  SE  learning,  and  to  develop  more 
valid  and  reliable  instruments  for  assessing  what  systems  engineers  need  to  learn,  need  to 
know,  and  need  to  do. 
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EXECUTIVE  SUMMARY 


This  research  focused  on  how  self-efficacy  might  be  used  as  a  means  to  assess  the 
competency  development  of  systems  engineering  (SE)  students.  The  method  was  also 
developed  as  a  first  step  in  attempting  to  predict  future  work  performance  of  systems 
engineers.  Self-efficacy,  as  it  relates  to  research  within  systems  engineering,  is  defined  as 
an  individual’s  belief  that  he  or  she  can  be  proficient  in  skills  necessary  to  become  a 
systems  engineering  professional.  SE  self-efficacy  is  also  defined  as  one’s  beliefs  in 
his/her  ability  to  leam  SE  knowledge,  skills  and  abilities  (KSAs).  The  research  is 
significant  as  current  studies  suggest  that  proficient  systems  engineers  are  expected  to  be 
able  to  integrate  and  apply  competencies  throughout  their  education,  professional 
development  and  experience.  The  thesis  presents  a  brief  introduction  to  the  Naval 
Postgraduate  School’s  (NPS’)  competency  model,  how  the  competency  model  relates  to 
the  SE  Competency  Survey,  an  in-depth  analysis  of  the  related  self-efficacy  research, 
methodology,  and  the  results  of  the  SE  Competency  Survey  instrument. 

The  study  detennined  how  self-efficacy  items,  identified  in  the  SE  Competency 
Survey  instrument,  relate  to  a  SE  competency  model  developed  at  NPS.  Systems 
engineering  competencies  were  identified  as  the  educational  learning  objectives  needed 
to  be  acquired  by  systems  engineers  to  perform  tasks  related  to  the  discipline.  The  KSAs 
were  grouped  into  cognitive  and  affective  domains,  As  such,  the  SE  Competency  Survey 
presents  the  cognitive  and  affective  KSAs  as  measures  of  performance.  The  SE 
Competency  Survey  instrument  is  based  on  a  subset  of  the  SE  learning  domains  that 
systems  engineering  students  are  expected  to  be  proficient  in  (systems  engineering, 
critical  thinking,  project  management  and  teamwork).  University  faculty  and  engineering 
organizations  can  use  the  instrument  to  assess  areas  of  skill  sets  that  students  and/or 
entry-level  engineers  are  strong,  weak  in,  or  need  to  develop  further. 

A  literature  review  led  the  team  to  believe  that  a  gap  exists  between  being  able  to 
assess  SE  KSA  and  measure  the  competency  of  students/entry  level  engineers  as  a  way  to 
predict  future  career  performance.  The  research  presents  an  assessment  of  the  roles  self- 
efficacy  and  cognitive/affective  skills  play  in  indicating  future  perfonnance  in  systems 
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engineering  related  careers.  The  role  of  self-efficacy  in  indicating  performance  in  SE 
related  careers  can  be  seen  as  that  of  a  psychosocial  construct.  It  is  influenced  by  one’s 
own  sense  of  capability  and  by  the  education,  training  and  support  received  by  others. 
The  NPS  research  team  cited  findings  by  the  National  Security  Leadership  Foundation, 
which  suggest  that  an  individual’s  level  of  self-efficacy  is  a  motivational  predictor  of 
how  well  a  person  will  perform  tasks  and  can  be  utilized  and  developed  to  leverage 
performance  enhancing  advantages;  an  individual’s  perceived  self-efficacy  greatly  affects 
his/her  ability  to  achieve  domain  specific  tasks.  Self-efficacy  serves  as  a  regulator  across 
a  broad  range  of  functioning,  as  the  key  detenninant  for  cognitive  and  affective 
processes.  Similarly,  the  research  is  partially  based  on  Prawat’s  finding  that  both 
cognitive  and  affective  skills  are  determined  to  be  vital  in  indicating  perfonnance  in  SE 
because  having  a  broad  range  of  KSAs  enables  knowledge  acquisition  and  application 
(1989).  The  NPS  team  considers  that  acquiring  domain  specific  knowledge  has  the 
advantage  of  allowing  one  to  facilitate  the  application,  or  performance,  of  work. 

The  study  of  self-efficacy  is  presented  with  a  detailed  explanation  of  the  statistical 
tests,  factor  analysis  (FA)  and  Cronbach’s  alpha,  which  were  used  to  assess  the  reliability 
of  the  survey  instrument.  The  FA  process  and  the  determination  of  Cronbach’s  alpha 
added  weight  to  the  reliability  of  the  SE  Competency  Survey  as  a  tool  to  assess  the 
perceived  self-efficacy  of  systems  engineering  students  to  perform  tasks  within  the  SE 
domains.  Current  research  findings  suggest  that  using  questionnaires,  such  as  the  SE 
Competency  Survey,  can  aid  in  the  identification  of  complex,  latent  psychological  traits — 
in  this  case,  self-efficacy.  Current  research  also  suggests  that  participants  in  a  study  such 
as  this,  are  not  likely  to  perceive  themselves  to  be  poor  in  a  competency  area  they  think  is 
very  important  (Moore  and  Rudd  2005,  76).  With  this  in  mind,  and  to  add  weight  to  the 
findings,  our  analysis  shows  that  Cronbach’s  alpha  was  determined  to  be  high — 
providing  support  that  the  NPS  survey  data  results  have  a  relatively  high  internal 
consistency  for  evaluation. 

To  provide  verification  of  the  research  method  used  to  measure  acquisition  of 
domain  specific  knowledge,  the  NPS  team  used  two  pre-  and  post-survey  renditions  of 
the  SE  Competency  Survey.  The  goal  of  the  survey  was  to  gauge  the  perceived  levels  of 


self-efficacy  among  systems  engineering  students,  over  time,  in  four  systems  engineering 
domains  as  they  continued  through  the  systems  engineering  program  at  NPS.  Survey 
items  were  written  with  the  intent  to  capture  domain  specific  skills  and  knowledge  sets 
within  critical  thinking,  systems  engineering,  teamwork  and  project  management.  The 
discussion  provided  describes  detailed  findings  on  the  results  of  the  survey  research  on 
each  of  the  four  domains,  as  well  as  commentary  on  possible  reasons  for  the  findings. 

In  general,  the  teamwork  results  indicate  mixed  findings  as  the  results  for  the  first 
cohort  did  not  show  significant  differences  in  the  average  response  rates  towards  many  of 
the  teamwork  aspects,  while  the  second  cohort  seems  to  have  improved  at  significant 
levels.  Among  the  first  cohort’s  results,  the  skill  was  observed  to  be  significant  at  p<.01, 
with  a  percent  increase  in  pre-  and  post-results  of  nine  percent.  The  second  cohort’s 
percent  increase  between  pre-  and  post-average  scores  was  30  percent,  and  was 
statistically  significant  at  p<.001.  A  possible  explanation  for  this  observation  is  that  lead 
faculty  incorporated  course  work  that  addressed  teamwork  into  the  curriculum  of  the 
second  cohort  ( RT19A ),  but  not  the  first  cohort.  This  was  because,  for  the  first  cohort,  it 
was  assumed  the  students  could  effectively  work  in  teams.  This  structural  method  may 
have  impacted  the  results.  This  finding  is  indeed  interesting  as  research  has  found 
teamwork  in  engineering  has  the  advantage  of  solving  complex  problems  (Ponton  et  al. 
2001,249). 

The  critical  thinking  data  analyses  of  the  first  cohort’s  results  seem  to  indicate 
that  the  self-efficacy  scores  in  critical  thinking  did  not  go  up  enough  to  exceed  rejection 
criteria  for  traditional  hypothesis  testing — which  would  contend  that  there  would  be  a 
significant  difference  between  pre-  and  post- averages.  All  but  two  of  the  skills  surveyed 
demonstrated  higher  averages  in  the  post-survey  results — meaning  that  even  though  the 
gains  were  not  significantly  different,  there  was  at  least  a  general  sense  of  improvement 
among  students.  An  analysis  of  the  second  cohort  yields  markedly  different  results,  with 
most  of  the  post-mean  differences  relating  to  critical  thinking  self-efficacy  skills 
statistically  significant  from  pre-survey  results  at  p<.001.  In  reviewing  the  critical 
thinking  results,  it  should  be  noted  that  the  structural  fonnat  in  teaching  the  first  cohort 
involved  referring  the  students  to  textbooks  on  the  topic  of  engineering  reasoning, 

xvii 


whereas  the  second  cohort  was  more  formally  taught  from  lectures  (i.e.,  through  parts  of 
course  work  and  seminars)  on  what  critical  thinking  is  and  how  it  is  used.  The  NPS 
research  team  feels  that  this  change  in  structure  may  have  impacted  the  survey  results. 

For  systems  engineering,  the  first  cohort’s  results  showed  significant  differences 
in  several  areas.  Two  skills  in  particular  showed  significant  differences  between  pre-  and 
post-results  at  p<.01.  These  were  the  student’s  ability  to  use  projected  warfighting 
capability  gaps  to  estimate  the  future  need  for  a  new  system  and  to  describe  what 
resources  will  be  necessary  to  conduct  tests  to  demonstrate  that  a  system  meets  its 
requirements.  One  self-efficacy  item  in  particular  reflected  statistically  significant  pre- 
and  post-survey  results  at  p<.001,  which  was  the  student’s  ability  to  pick  a  technology  for 
a  system  considering  that  the  technology  readiness  level  is  estimated,  and  uncertain. 
There  were  significant  differences  in  means  observed  in  all  self-efficacy  items  relating  to 
systems  engineering.  These  results  may  be  due  to  the  change  in  structural  format  of 
instruction  delivery  between  that  of  the  first  and  second  cohort.  The  first  cohort 
integrated  their  authentic  project  within  each  of  the  courses  in  the  curriculum.  The  second 
cohort  had  three  additional  consecutive  project  courses  added  to  their  curriculum  in 
which  the  instructor  reviewed  all  SE  methods  and  processes  in  the  context  of  the  project. 
This  facilitated  near  transfer  of  the  learning  of  SE  methods  and  processes  to  the  authentic 
project  context.  The  NPS  team  feels  that  this  approach  to  teaching  systems  engineering 
concepts  in  context  of  authentic -learning  projects  seems  to  improve  self-efficacy  and  is 
more  effective  in  teaching. 

For  project  management,  most  of  the  first  cohort’s  results  did  not  go  up  enough  to 
exceed  traditional  rejection  criteria  for  hypothesis  testing.  An  exception  to  this 
observation  was  in  the  skill  of  estimating  quantitatively  the  likelihood  and  relative  impact 
of  events  on  a  development  project  that  represent  the  most  pressing  risks,  which  had  an 
increase  of  1 1  percent,  and  the  difference  in  mean  scores  was  statistically  significant  at 
p<.05.  Results  of  the  second  cohort’s  survey  items  relating  to  project  management 
showed  statistically  significant  differences  in  the  mean  scores  at  p<.001 — exceeding 
rejection  criteria  for  traditional  hypothesis  testing.  Similar  to  the  analysis  discussed  on 
systems  engineering,  the  project  management  results  may  be  due  to  the  change  in 
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structural  format  of  instruction  delivery  between  that  of  the  first  and  second  cohort.  As  in 
the  case  in  the  systems  engineering  domain,  the  second  cohort  was  explicitly  taught  how 
SE  skills  relate  to  the  project  via  learning  activities.  The  NPS  team  feels  that  assisting 
them  in  near  transfer  of  knowledge  and  ability  to  their  project  may  have  influenced  the 
survey  results  to  reflect  an  increase  in  self-efficacy. 

Results  from  the  survey  analysis  led  to  the  conclusion  that  self-efficacy  in  systems 
engineering  can  be  reasonably  assumed  to  be  positively  affected  by  a  rigorous 
educational  program  and  training.  As  such,  the  NPS  team  feels  that  the  SE  Competency 
Survey  can  be  used  by  university  faculty  and  engineering  enterprises  to  measure  self- 
efficacy  in  systems  engineers  in  an  overall  effort  to  predict  future  career  perfonnance. 
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I.  INTRODUCTION 


Recent  studies  into  career  development  models  have  led  to  the  general  finding 
that  contemporary  firms  are  searching  for  well-rounded  individuals,  who  possess  both 
technical  and  non-technical  skills  (Joshi  et  al.  2010,  3).  Scholarly  studies  examining 
hiring  practice  trends  through  content  analysis  of  job  advertisements  and  job  types,  have 
observed  that  stakeholders  prefer  to  employ  well-rounded  employees  with  business 
knowledge,  interpersonal  skills  and  technical  skills  (Joshi  et  al.  2010,  3).  According  to 
Joshi  et  al.,  “Students  gain  their  awareness  of  the  importance  of  technology  related  job 
skills  from  the  academics  who  teach  them”  (2010,  3).  As  such,  a  goal  of  university 
curricula  and  systems  engineering  enterprises  should  be  to  educate  to  provide  the 
necessary  technical  and  non-technical  background  to  enable  students  to  become 
proficient  in  performing  systems  engineering  competencies.  Systems  engineering  (SE) 
competencies  are  defined  as  the  knowledge,  skills  and  abilities  (KSAs)  necessary  to 
perform  tasks  related  to  the  discipline.  Proficient  systems  engineers  are  expected  to  be 
able  to  integrate  and  apply  such  competencies  throughout  their  education,  professional 
development  and  experience. 

Pertinent  to  our  goal  as  an  academic  institution  to  educate  students  towards 
becoming  proficient  engineers  was  the  need  to  identify  the  current  KSAs  major 
stakeholders  seek  in  the  employment  of  systems  engineers.  When  considering  the 
development  of  key  technical  and  non-technical  KSAs,  we  also  examined  the  role  of  self- 
efficacy  as  a  regulator  of  cognitive  and  affective  learning  processes,  as  studies  have 
shown  perceived  self-efficacy  contributes  to  cognitive  and  affective  development  and 
functioning  (Bandura  1993,  1). 

This  thesis  describes  the  process  that  was  undertaken  by  the  Systems  Engineering 
Department  at  the  Naval  Postgraduate  School  (in  Monterey,  California),  to  measure  and 
evaluate  the  level  of  competency,  or  “self-efficacy,”  perceived  by  systems  engineering 
students.  To  obtain  an  understanding  of  where  graduate  systems  engineering  students 
stand  as  far  as  developing  the  necessary  competencies  they  need  to  be  successful  systems 

engineers,  we  developed  a  series  of  surveys  to  gauge  their  level  of  self-efficacy. 
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Systems  engineering  self-efficacy,  in  this  study,  is  defined  as  the  individual’s 
belief  that  he  or  she  can  be  proficient  in  skills  necessary  to  become  a  systems  engineering 
professional.  System’s  engineering  self-efficacy  also  refers  to  one’s  beliefs  in  his/her 
ability  to  leam  systems  engineering  skills. 

Self-efficacy  can  be  further  defined  as  a  “psychological  characteristic  known  for 
its  ability  to  have  an  enduring  influence  on  related  behavior  and  interests”  (Mau  2003, 
234).  Expanding  on  this  description,  self-efficacy  has  also  been  described  as  a  formative 
social  characteristic  which  can  have  a  spiral  effect,  in  that  once  it  is  well  established  in  an 
individual,  it  can  influence  that  individual  to  make  choices  and  attempt  tasks  that  further 
strengthen  and  increase  confidence  in  his  or  her  abilities  to  perform  tasks  within  a  domain 
(Lucas  and  Cooper  2004,  12).  An  objective  of  this  study  was  to  detennine  whether  self- 
efficacy,  alongside  being  a  psychological  trait,  can  be  taught,  or  enhanced,  through 
education.  Research  suggests  that  when  looking  at  self-efficacy,  particularly  in 
engineering  students,  it  is  important  for  an  engineering  professor  to  understand  how  self- 
efficacy  relates  to  learning  concepts. 

The  context  of  how  engineering  students  leam  is  often  categorized  using  Bloom’s 
taxonomy.  Engineering  education  involves  learning  objectives,  which  are  typically 
organized  around  Bloom’s  taxonomy  of  cognitive  and  affective  processes.  Cognitive  and 
affective  processes  within  Bloom’s  taxonomy  refer  to  levels  of  observable  actions  which 
indicate  learning  is  occurring.  The  cognitive  domain  involves  knowledge  and  the 
development  of  intellectual  skills,  whereas  the  affective  domain  deals  with  the 
motivations  and  attitudes  involved  in  learning  (Clark  1999). 

In  keeping  engineering  learning  objectives  in  mind,  it  takes  a  great  amount  of 
effort  on  behalf  of  the  professor  to  use  mastery  experiences,  vicarious  experiences,  verbal 
persuasion,  and  physiological/emotive  arousal  strategies  within  instruction,  to  improve 
self-efficacy  (Ponton  et  al.  2001,  249).  As  such,  we  hope  findings  from  this  study  can  be 
used  to  gauge  self-efficacy  in  systems  engineering  students  and  to  implement  changes  to 
improve  systems  engineering  instructional  delivery  strategies. 
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The  thesis  begins  with  a  discussion  on  the  methodology  that  was  utilized  to 
acquire  and  analyze  data  necessary  to  evaluate  the  level  of  self-efficacy  of  systems 
engineering  students  to  independently  perform  systems  engineering  tasks.  The  review 
includes  a  discussion  on  the  survey  methods  used  to  study  self-efficacy  in  systems 
engineering,  which  included  the  process  of  utilizing  the  statistical  method  of  rotated 
factor  analysis  as  a  means  to  test  for  the  validity,  or  effectiveness,  of  the  survey 
instrument  to  capture  reliable  data. 

The  goal  of  the  survey  was  to  gauge  the  perceived  levels  of  self-efficacy  among 
systems  engineering  students,  over  time,  in  four  systems  engineering  domains  as  they 
continued  through  the  systems  engineering  program  at  NPS.  Survey  items  were  written 
with  the  intent  to  capture  domain  specific  skills  and  knowledge  sets  within  critical 
thinking,  systems  engineering,  teamwork  and  project  management.  The  discussion 
provided  describes  detailed  findings  on  the  results  of  the  survey  research  on  each  of  the 
four  domains,  as  well  as  commentary  on  possible  reasons  for  the  findings.  Results  from 
the  survey  analysis  led  to  the  conclusion  that  self-efficacy  in  systems  engineering  can  be 
reasonably  assumed  to  be  positively  affected  by  a  rigorous  educational  program  and 
training. 

A.  RESEARCH  QUESTIONS 

The  background  of  this  topic  has  presented  several  issues  that  were  considered  in 
the  formulation  of  the  research  questions.  The  following  research  questions  have  been 
developed  in  response  to  the  previously  outlined  research  objectives.  These  research 
questions  will  guide  the  path  of  this  research  that  will  culminate  in  their  answers  and 
conclusions  provided  in  Chapters  IV  and  V. 

1.  Primary  Research  Question 

The  primary  research  question  will  help  determine  whether  self-efficacy  can  be 
reliably  measured  to  validate  the  attainment  of  domain  specific  skills  among  systems 
engineering  students. 

•  Primary  research  question  1 :  Can  self-efficacy  be  used  as  an  indicator  of 
systems  engineering  competency  level  achievement? 
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2.  Secondary  Research  Questions 

While  examining  the  possible  methods  to  reliably  measure  and  assess  self- 
efficacy  as  a  competency  construct,  a  set  of  secondary  research  questions  developed. 

•  Secondary  research  question  1:  What  is  self-efficacy  in  systems 
engineering? 

•  Secondary  research  question  2:  How  might  self-efficacy  be  used  in 
systems  engineering? 

•  Secondary  research  question  3:  What  methods  can  be  used  to  assess  self- 
efficacy?  What  methods  can  be  used  in  systems  engineering? 

•  Secondary  research  question  4:  Can  a  survey  instrument  be  used  to 
identify  achievement  of  specific  knowledge,  skills  and  abilities  (KSAs)  as 
indicators  of  competency  in  performing  systems  engineering  tasks? 

•  Secondary  research  question  5:  If  possible  within  time,  participant,  and 
resource  constraints,  can  the  systems  engineering  survey  instrument  be 
used  by  other  engineering  disciplines  as  a  model  to  implement  the  self- 
efficacy  based  method,  and  can  it  be  tailored  to  meet  an  organization’s 
requirements  with  minimal  effort/changes? 

•  Secondary  research  question  6:  What  roles  do  self-efficacy  and 
cognitive/affective  skills  play  in  indicating  performance  in  systems 
engineering  related  careers? 

B.  PURPOSE/BENEFIT 

The  benefit  of  the  study  will  be  two-fold.  First,  a  direct  result  of  the  research  will 
be  a  survey  instrument,  which  can  be  used  to  reliably  measure  self-efficacy  as  it  relates  to 
competency  levels  in  systems  engineering.  The  research  will  serve  to  test  and  verify  the 
usefulness  of  the  self-efficacy  survey  instrument  in  determining  levels  of  competency 
achievement.  Second,  the  survey  instrument  will  be  a  tool  that  naval  system  commands 
and  educational  institutions  can  use  to  measure  and  verify  the  level  of  competency  their 
employees/students  have  achieved.  This  can  then  be  used  by  organizations  to  track 
workforce  development,  to  confidently  assign  work  tasks,  to  assess  the  ability  to  perfonn 
standard  systems  engineering  tasks,  and  to  create  individual  development  plans  for  career 
management. 
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C.  THESIS  STATEMENT 

Our  research  maps  competencies  as  theory  based  indicators  of  self-efficacy  to 
establish,  with  confidence,  which  measures  of  graduate  level  education  would  be 
expected  to  indicate  higher-levels  of  SE  self-efficacy. 

D.  RESEARCH  METHODS 

The  research  methods  discussed  in  the  following  chapters  will  discuss  and 
describe  the  process,  methods  and  products  developed  by  NPS  to  answer  the 
aforementioned  research  questions.  The  primary  NPS  research  product  was  a  survey 
instrument,  which  was  designed  to  gather  information  from  students  on  self-efficacy. 
Specifically,  the  survey  was  given  over  a  period  of  two  years  to  two  different  cohorts. 
The  research  methods  used  to  examine  the  reliability  of  the  survey  instrument  to  measure 
self-efficacy  as  a  latent  psychological  construct  included  factor  analysis  and  Cronbach’s 
alpha. 

E.  CHAPTER  SUMMARY 

In  this  chapter  an  introduction  to  the  research  topic  was  presented  along  with  a 
stakeholder  analysis  on  the  needs  of  industry,  which  concluded  that  there  is  a  need  for 
engineers  who  are  competent  in  both  technical  and  non-technical  skills.  The  introduction 
followed  with  a  description  of  self-efficacy,  and  how  it  relates  to  systems  engineering. 
The  first  chapter  concluded  with  a  brief  introduction  to  the  survey  instrument  and 
methodology  used  to  conduct  the  research  in  an  effort  to  answer  the  research  questions, 
which  were  also  provided.  Finally,  the  purpose  and  benefit  of  the  research  was  stipulated 
to  be  two-fold.  The  research  led  to  the  development  of  a  survey  instrument,  which  can  be 
used  to  reliably  correlate  self-efficacy  to  competency  levels  in  systems  engineering. 
Alongside  the  SE  competency  career  development  model,  the  survey  instrument  was 
suggested  as  a  resource  whereby  organizations  can:  track  workforce  development,  assign 
work  tasks,  assess  the  ability  of  engineers  to  perfonn  standard  SE  tasks,  and  create 
individual  development  plans  for  career  management. 
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II.  LITERATURE  REVIEW 


The  following  is  a  brief  literature  review  of  past  and  current  research  on  self- 
efficacy  constructs,  and  in  particular  addresses  literature  that  supports  the  importance  of 
self-efficacy  and  its  influence  on  academic  achievement  and  work  perfonnance.  The 
review  suggests  relevant  connections  can  be  made  between  self-efficacy  and  social 
cognitive  theory  (SCT),  entrepreneurship  (Lucas  et  al.  2009),  development  of  leadership 
and  technical  skills  (Joshi  et  al.  2010),  and  meta-cognition  (Lang  2012).  Metacognition, 
the  knowledge  of  one’s  own  cognition  and  the  regulation  of  such  skills,  has  been  found  to 
be  an  important  factor  in  successful  task  perfonnance  (Kraiger,  Ford  and  Salas  1993). 

Several  studies  extending  self-efficacy  theory  have  related  it  to  vocational 
behavior,  suggesting  expectations  of  self-efficacy  aid  in  the  determination  of  career 
decisions,  can  serve  as  predictors  of  achievements  of  men  and  women,  and  may  be 
particularly  useful  in  understanding  women’s  career  development  (Hackett  and  Betz 
1981).  However,  the  focus  of  our  discussion  is  a  literature  review  of  the  lessons  that  can 
be  gleaned  from  research  on  the  importance  of  self-efficacy  in  indicating  a  predilection 
towards  successful  work  performance. 

A.  SELF-EFFICACY  IN  RELATIONSHIP  TO  SOCIAL  COGNITIVE 

THEORY 

According  to  Lent  et  al.,  “In  a  review  of  Social  Cognitive  Theory  (SCT)  and 
career  and  academic  interests,  Lent  et  al.  (1994)  provide  evidence  that  self-efficacy  in  a 
particular  domain  helps  develop  the  individual’s  initial  interest  in  a  corresponding  career 
area,  followed  by  the  selection  of  career  paths,  and  then  supports  the  higher  levels  of  both 
academic  perfonnance  and  persistence  in  those  domains”  (2009,  740).  We  argue  that 
such  “higher  levels”  of  academic  perfonnance  are  indicative  of  the  development  of 
cognitive  and  affective  processes,  which  we  have  identified  via  the  NPS  competency 
model,  as  discussed  in  Chapter  III,  on  the  methodology  and  results  of  the  NPS  research. 

SCT  posits  that  human  agency  (defined  here  as  intentional  behavior)  is  affected 
by  the  “reciprocal  interaction  between  personal  behavior,  internal  personal  factors  (i.e., 
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one’s  cognitive,  biological,  and  affective  characteristics)  and  the  environment  (i.e., 
everything  external  to  the  individual)  where  all  three  determinants  interact  with  various 
magnitudes  of  influence  that  is  context  dependent”  (Ponton  et  al.  2001,  247). 

Foundational  research  performed  by  Lucas  et  al.  (2009),  summarizes  the  role  self- 
efficacy  has  in  enabling  students  to  pursue  careers  involving  innovation.  Lucas  et  al. 
suggest,  “based  on  Social  Cognitive  Theory,  one  would  expect  that  individuals  with  a 
high  self-efficacy  in  a  particular  domain  both  perform  work  tasks  better,  and  are  more 
innovative  in  the  way  in  which  they  go  about  their  work  in  that  domain”  (2009,  740). 
They  suggest  that  self-efficacy  is  central  to  career  choice  and  in  accepting  careers  in  more 
challenging  fields,  particularly  in  science  and  engineering.  Furthennore,  Lucas  et  al.  cite 
Bandura’s  findings  that  suggest  that  domain  self-efficacy  (in  this  context,  the  confidence 
individuals  have  in  their  ability  to  undertake  a  range  of  activities  within  a  specific 
domain)  applies  to  career  selection. 

1.  Work  Performance 

In  referencing  Stajkovic  and  Luthans  (1998),  Lucas  et  al.  contend  that  the  role  of 
self-efficacy  continues  to  be  important  once  individuals  have  entered  their  careers, 
predicting  better  work  perfonnance  (Lucas  et  al.  2009,  740).  In  addition,  they  go  on  to 
explain,  “Causal  inferences  about  the  effect  of  self-efficacy  on  work  performance  are 
supported  by  experimental  research  that  uses  random  group  assignment  and  the 
manipulation  of  self-efficacy”  (Lucas  et  al.  2009,  740).  Lucas  et  al.  (2010,  738)  have 
explored  the  correlation  between  various  workplace  experiences  and  venturing  and 
technology  self-efficacy.  They  also  focus  heavily  on  the  development  of  self-efficacy 
through  authentic  work  experience. 

For  work  experience  to  increase  self-efficacy  for  future  entrepreneurial  and 
technical  applications,  it  must  be  authentic  (Lucas  et  al.  2009,  741).  Lucas  et  al.  define 
authentic  as  being  seen  by  the  students  as  having  required  similar  skills  and  levels  of 
performance  that  they  believe  will  be  expected  of  them  if  they  pursue  innovative  careers 
in  engineering  (2009,  741).  They  also  contend  that  if  self-efficacious  individuals  set 
themselves  more  challenging  goals  and  are  prepared  to  take  on  unfamiliar  tasks  then,  as 
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previously  mentioned,  self-efficacy  is  expected  to  greatly  influence  the  likelihood  that  the 
individuals  will  attempt  innovative  behavior  (Lucas  et  al.  2009,  740).  Their  research 
measures  two  types  of  self-efficacy,  one  for  venturing  skills  (starting  one’s  own  business; 
entrepreneurial)  and  the  other  for  technology.  They  then  map  the  factors  in  social 
cognitive  theory  that  predict  higher  self-efficacy  to  the  elements  of  undergraduate  work 
experience. 

Lucas  et  al.  (2009)  achieved  the  aforementioned  objectives  via  a  survey 
methodology,  which  we  aimed  to  emulate  in  our  study.  Their  model  and  research 
examined  the  effects  of  IT  skills  self-efficacy  and  IT  importance  on  student’s  attraction  to 
IT  careers.  A  key  difference  between  our  research  emphases  is  that  Lucas  et  al.  discuss 
and  describe  implications  of  self-efficacy  and  background  characteristics  of  survey 
participants,  types  of  industry  placement  and  the  university  the  students  are  drawn  from, 
in  a  test  for  possible  sampling  bias.  We  did  not  ask  for  student  demographic  and 
background  data  in  an  effort  to  encourage  honest  responses,  and  as  such,  do  not  have  a 
valid  means  to  match  student  demographic  and  background  data  with  responses  for  any 
correlation  analysis. 

The  Lucas  et  al.  (2009)  self-efficacy  survey  items  intended  to  measure  venturing 
and  technical  self-efficacy  in  undergraduate  engineering  students  who  had  previous  work 
placements.  Their  survey  also  included  questions  that  characterized  different  elements  of 
work  experiences,  including  the  orientation  of  the  company  at  which  the  students  worked, 
the  nature  and  difficulty  of  assigned  work,  and  the  presence  of  role  models  where  the 
students  were  placed  (Lucas  et  al.  2009,  739).  Of  particular  interest  to  our  study  was  the 
survey  scale,  which  asked  students  to  rate  their  self-confidence  in  perfonning  science  and 
technology  related  tasks,  as  a  measure  of  engineering  competency.  The  survey  instrument 
scale  ranged  from  0  (not  at  all  confident)  to  10  (completely  confident).  Items  were 
written  to  measure  self-confidence  in  performing  tasks  undergraduates  would  be  expected 
to  perform  if  they  were  to  be  proficient  in  technology  related  work.  These  skills  and 
abilities  included  understanding  the  limits  of  technology,  translating  user  requirements 
into  technical  requirements,  and  leading  a  team  of  technical  professionals. 
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Lucas  et  al.  (2009)  then  used  regression  analysis  to  examine  correlations  among 
student  background,  organizational  and  other  predictor  variables.  The  results  of  their 
survey  data  analysis  led  to  the  conclusion  that  on  average,  work  experience  had  little 
effect  on  student  self-efficacy.  Yet,  when  the  survey  data  reflected  that  the  student 
experiences  had  the  qualities  generally  accepted  to  predict  enhanced  self-efficacy,  strong 
differences  were  observed.  Lucas  et  al.  suggest  that  these  generally  accepted  constructs 
are:  authentic  experience,  social  influence,  vicarious  experience  and  emotional  states 
(2009).  Their  results  indicate  that  authenticity,  feedback  on  performance,  and  how  well 
students  feel  they  have  performed,  are  the  dominant  predictors  of  self-efficacy  (2009). 

Lucas  et  al.  explain,  “The  effect  of  workplace  experience,  generally  found  in 
literature  to  be  important,  is  widely  viewed  by  university  instructors  as  having  a 
consequential  impact  on  student  readiness  for  the  world  of  work”  (2009,  739).  Lucas  et 
al.  stipulate  that  one  approach  to  prepare  university  students  for  innovative  technical 
careers  is  to  emphasize  methods  of  education  that  replicate  aspects  of  real  work 
experience  (2009,  739).  They  also  very  astutely  point  out  that  a  “major  task  for  higher 
education  is  the  development  of  employees  who  not  only  have  the  right  skills  and 
attitudes,  but  also  the  ability  to  leam  from  experience  and  adapt  within  a  dynamic  and 
rapidly  changing  environment”  (Lucas  et  al.  2009,  739).  The  role  of  research  should  be  to 
determine  what  aspects  of  a  student’s  experience  enhance  self-efficacy  for  venturing  and, 
independently,  self-efficacy  for  technology  (Lucas  et  al.  2009,  739).  Lucas  et  al.  state, 
“Pinquart  et  al.  (2003)  suggest  that  increased  knowledge  and  understanding  of  the 
industry  environment  as  a  workplace  helps  to  ease  the  transition  from  education  to 
employment”  (2009,  740).  We  argue  that  the  use  of  our  survey  instrument  can  aid  in  the 
identification  of  where  students  are  at  in  gaining  the  required  knowledge  and 
understanding  of  industry  practices  to  be  proficient  engineers.  The  NPS’s  competency 
development  model  can  help  to  identify  at  which  career  developmental  stage  students 
should  gain  the  knowledge,  which  would,  we  assert,  ease  the  transition  from  their 
education  to  subsequent  work  assignments.  Research  by  Lucas  et  al.  (2009)  does  not 
answer  whether  or  not  this  requires  a  higher  level  of  cognition,  our  research  fills  this  gap. 
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Furthermore,  we  contend  that  our  research  aids  in  the  identification  of  the  characteristic 
experiences  and  knowledge  students  need  to  strengthen  self-efficacy  in  technical  and 
non-technical  fields. 

B.  SELF-  EFFICACY:  PREDICTOR  OF  WORK  PERFORMANCE 

Stajkovic  and  Luthans  (1998)  surveyed  114  research  reports  on  self-efficacy  and 
work-related  performance  and  found  that  it  consistently  predicts  higher  levels  of  work 
performance  across  various  types  of  organizations  and  activities.  They  researched  the 
correlation  between  self-efficacy  and  work-related  performance,  hypothesizing  that  there 
is  an  overall  positive  relationship  between  self-efficacy  and  work-related  perfonnance 
(Stajkovic  and  Luthans  1998,  241).  A  meta-analysis  (empirical  research  examining  the 
results  of  all  114  studies)  performed  by  the  team  aggregated  and  assessed  individual 
research  findings  pertaining  to  the  relationship  between  self-efficacy  and  work-related 
performance  (Stajkovic  and  Luthans  1998,  240).  In  their  meta-analysis,  they  investigate 
the  overall  magnitude  of  the  relationship  between  the  two  constructs  (Stajkovic  and 
Luthans  1998,  240). 

Their  findings  indicate  that  a  variable  of  potential  moderating  importance 
regarding  the  relationship  between  self-efficacy  and  work-related  performance  is  task 
complexity,  and  that  past  research  on  self-efficacy  has  shown  it  is  related  to  several 
work-perfonnance  measures.  Such  measures  are  “adaptability  to  advanced  technology, 
managerial  idea  generating/perfonnance,  skill  acquisition,  newcomer  adjustment  to  an 
organizational  setting  and  naval  performance  at  sea”  (Stajkovic  and  Luthans  1998,  240). 

1.  Self-Efficacy:  Predictor  of  Academic  Performance  and  Perceived 
Career  Options 

Lent  et  al.  (1984)  tested  their  hypothesis  that  efficacy  expectations  should  relate 
to  persistence  and  success  in  pursuing  educational  and  career  goals.  Their  findings 
suggested  that  students’  beliefs  about  their  ability  to  complete  the  educational 
requirements  of  various  engineering  and  science  fields  were  indicative  of  subsequent 
academic  perfonnance.  For  instance,  students  reporting  comparatively  strong  self- 
efficacy  were  generally  found  to  achieve  higher  grades  and  were  much  more  likely  to 
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persist  in  technical/scientific  majors  over  a  one-year  period  compared  to  those  with  low 
self-efficacy.  Subsequent  research  by  Lent  et  al.  (1986)  served  to  extend  results  of  other 
studies  (Hackett  and  Betz  1981)  by  assessing  the  relation  of  efficacy  beliefs  to  perceived 
career  options  in  science  and  engineering  students.  In  particular,  they  studied  the 
applicability  of  self-efficacy,  alongside  “ability,  achievement,  and  interest  measures  in 
predicting  academic  success,  persistence,  and  range  of  perceived  career  options  in 
technical/scientific  fields”  (Lent  et  al.  1986,  265). 

Self-efficacy  was  assessed  via  survey  instruments  that  measured  self-efficacy, 
career  indecision,  self-esteem,  expressed  vocational  interests,  and  range  of  perceived 
vocational  options  in  technical/scientific  fields  (Lent  et  al.  1986).  Similar  to  our  mode  of 
survey  delivery,  their  surveys  were  administered  as  pre  and  post  surveys,  specifically 
during  the  first  and  final  class  sessions.  Self-efficacy  was  assessed  by  asking  subjects  to 
indicate  whether  they  believed  they  could  successfully  perfonn  the  educational 
requirements  (ER)  and  expected  job  duties  in  15  science  and  engineering  fields. 
Individual  levels  of  self-efficacy  (ER-L)  scores  were  obtained  by  summing  the  number  of 
fields  subjects  believed  they  could  complete.  Strength  of  self-efficacy  (ER-S)  was 
assessed  by  having  subjects  estimate  their  degree  of  confidence  in  their  ability  to 
complete  the  ER  and  job  duties.  Similar  to  our  study,  self-efficacy  to  perform  a  task  was 
indicated  using  a  numerical  scale.  Whereas  our  scale  was  0  to  100,  with  0  representing  no 
confidence  and  100  representing  complete  confidence,  Lent  et  al.  used  a  range  of  one  to 
ten,  from  completely  unsure  (1)  to  completely  sure  (10)  (Lent  et  al.  1986,  266).  Then, 
strength  scores  for  each  subject  were  calculated  by  dividing  the  summed  strength 
estimates  by  15,  the  total  number  of  major/career  fields  (Lent  et  al.  1986,  266).  It  is 
noteworthy  to  emphasize  here  that  again,  similar  to  our  study,  students  were  asked  to  rate 
their  ability  to  perform  specific  accomplishments  considered  to  be  critical  to  academic 
success  and  performance  in  science  and  engineering  fields. 

Subsequent  findings  by  Lent  et  al.  concluded  that  student’s  beliefs  about  their 
ability  to  complete  the  educational  requirements  of  various  science  and  engineering  fields 
were  predictive  of  subsequent  academic  performance  (1984,  265).  Furthermore, 
hierarchical  regression  analyses  indicated  that  “self-efficacy  contributed  signification 
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unique  variance  to  the  prediction  of  grades,  persistence,  and  range  of  perceived  career 
options  in  technical/scientific  fields”  (Lent  et  al.  1984,  265). 

Thus,  self-efficacy  has  been  shown  to  be  a  useful  measure  in  the  prediction  of 
academic  performance,  yet  there  is  a  lack  of  research  to  support  the  assertion  on  the 
usefulness  of  self-efficacy  as  a  reliable  indicator  in  relation  to  competencies  that  define 
career  perfonnance.  Our  research  aims  to  close  this  gap. 

C.  RESEARCH  ON  SELF-EFFICACY  IN  INFORMATION 

TECHNOLOGY— A  PARALLEL  TO  SYSTEMS  ENGINEERING 

Similar  to  SE  self-efficacy,  infonnation  technology  (IT)  self-efficacy  refers  to 
one’s  beliefs  about  his/her  ability  to  learn  IT  skills  necessary  to  conduct  IT  work  (Joshi  et 
al.  2010,  2).  High  self-efficacy  for  a  domain  such  as  IT  would  be  theorized  to  lead  to 
choice  (versus  avoidance)  of  IT  coursework  and  college  majors,  to  facilitate  performance 
on  IT  related  tasks,  and  to  lead  to  persistence  during  difficult  academic/career  related 
experiences  (Joshi  et  al.  2010,  2).  As  Joshi  et  al.  explain,  “Students  gain  their  awareness 
of  the  importance  of  IT  job  skills  from  referent  others — including  the  academics  who 
teach  them,  peers  who  socialize  with  them,  and  parents  who  guide  them”  (2010,  3).  This 
sense  of  IT  skills  importance  refers  to  one’s  beliefs  about  the  value  of  IT  skills  necessary 
to  be  successful  in  the  IT  profession  (Joshi  et  al.  2010,  2).  Our  research  does  not  correlate 
a  student’s  perceived  level  of  importance  of  SE  skills — perhaps  this  could  be  a  future 
area  of  research. 

Similar  to  Lucas  et  al.,  Joshi  et  al.  analyzed  self-efficacy  survey  responses  of 
university  students  with  the  intent  to  study  gender  differences  (2010,  2).  Joshi  et  al. 
(2010)  conducted  a  three-part  study  that  aimed  to  enhance  the  overall  understanding  of 
attitudes  about  the  IT  field  and  subsequent  interest  in  an  IT  career.  The  significance  of 
their  research,  as  it  relates  to  the  NPS  study  on  SE  self-efficacy,  is  that  both  research 
teams  are  aiming  to  study  the  dynamics  between  self-efficacy  in  technical  fields.  So,  for 
the  purposes  of  NPS’ s  research,  their  findings  are  particularly  relevant  and  noteworthy. 

Phase  I  of  the  Joshi  et  al.  project  involved  an  analysis  of  three  categories  of  IT  job 
skills  (technical,  human  and  business),  across  three  genres  of  texts:  scholarly  articles, 
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practitioner  literature  and  online  job  ads  (2010,  2).  Likewise,  in  a  study  of  employment 
opportunities  for  recent  college  graduates,  Wong,  von  Hellens  and  Orr  used  both  IT  job 
ads,  surveys  and  interviews  with  stakeholders  and  determined  that  non-technical  skills  are 
increasingly  more  important  than  technical  skills  (Joshi  et  al.  2010,  3).  The  Phase  II  study 
revealed  that  while  masculine  stereotypes  are  applied  to  most  technical  skills,  and 
feminine  stereotypes  to  most  interpersonal  skills,  gender  neutral  skills  (i.e.,  initiative, 
ability  to  work  under  pressure,  critical  thinking,  and  problem  solving)  have  emerged  as 
important  constructs  today’s  students  must  possess.  Phase  III  of  their  study  focused  on 
the  development,  validation  and  effects  of  IT  related  self-efficacy  and  IT  skills 
importance  on  IT  career  intentions  (Joshi  et  al.  2010,  2).  To  achieve  this  objective,  Joshi 
et  al.  used  a  survey  methodology  similar  to  Lucas  et  al.  and  NPS  research.  The  Joshi  et  al. 
survey  asked  students  to  specify  their  level  of  self-confidence  in  their  perceived  ability  to 
“learn  and  engage”  in  36  IT  skill  items.  Results  were  subjected  to  factor  analysis,  to 
uncover  the  underlying  factor  structure  of  the  IT  self-efficacy  construct.  Two  constructs 
emerged  from  the  factor  analysis  results:  non-technical  and  technical  self-efficacy.  The 
significance  of  the  research  of  Joshi  et  al.,  particularly  within  Phase  III,  is  that  it  is 
another  example  of  a  successful  attempt  at  measuring  a  latent  psychological  trait  using  a 
survey  model.  It  adds  weight  to  the  idea  that  task  specific  measures  in  the  form  of  a 
survey,  might  be  superior  to  other  approaches  to  self-efficacy  assessment  (i.e.,  ER) 
(Hackett  and  Betz  1984). 

1,  Stakeholder  Analysis 

According  to  Joshi  et  al.,  “The  process  of  consulting  with  stakeholders  to  solicit 
feedback  regarding  the  importance  of  skills  and  the  quality  of  training  that  universities 
provide,  ensures  that  education  meets  the  needs  of  industry”  (2010,  2).  Stakeholder 
analyses  assessing  the  need  for  technically  competent  professionals  in  industry  have  led 
to  the  observation  of  the  shifts  in  the  composition  of  the  IT  workforce  and  accompanying 
changes  in  the  relative  importance  of  IT  skills  (Joshi  et  al.  2010,  3).  For  instance: 

low  skilled  IT  job  positions  (such  as  data  entry  and  computer  operators) 
are  disappearing.  IT  jobs  are  shifting  from  traditional,  centralized  MIS 
[management  infonnation  systems]  divisions  out  to  business  units, 
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necessitating  employees  to  have  more  business  knowledge  and  stronger 

interpersonal/management  skills  for  working  with  clients.  (Joshi  et  al. 

2010,3) 

Joshi  et  al.  note,  “In  addition  to  studies  that  examine  the  perceptions  of 
stakeholder  groups,  scholars  have  investigated  shifts  in  IT  skills  demand  and  job  types 
through  content  analysis  and  job  advertisements”  (2010,  3).  Findings  have  suggested  that 
systems  analysts  mostly  value  their  non-technical  roles  and  skills,  whereas  systems  users 
allocate  more  significance  to  the  analyst’s  technical  skills  (Joshi  et  al.  2010,  3).  In  a 
similar  study  of  webmasters  and  users,  it  was  found  that  users  valued  the  importance  of 
technical  skills,  whereas  webmasters  asserted  organizational  skills  were  more  important 
to  their  successful  job  performance  (Joshi  et  al.  2010,  3). 

McGee  suggests  that  the  lack  of  non-technical  skills  will  result  in  a  failure  of  an 
IT  professional  to  be  promoted  to  a  manager  position  (Joshi  et  al.  2010,  3).  Stakeholder 
analyses  also  indicate  that  “the  high  failure  rate  of  IT  projects  is  not  attributed  to  the 
technological  complexities  of  projects  or  the  technical  incompetency  of  the  project  teams. 
Rather,  it  is  attributed  to  the  non-technical  skills  within  the  IT  project  management 
teams”  (Joshi  et  al.  2010,  3). 

D.  ENHANCING  SELF  EFFICACY  AND  COGNITIVE  MOTIVATION 

Literature  suggests  that  the  presence  of  self-efficacy  helps  to  motivate  students  to 
delve  into  engineering  course  work,  which  aids  in  their  development  as  a  capable 
professional  (Ponton  et  al.  2001,  249).  Sources  of  self-efficacy  evaluations  are  mastery 
experiences,  vicarious  experiences,  verbal  persuasion  and  physiological/emotive  states 
(Ponton  et  al.  2001,  249).  Ponton  et  al.  point  out  that  curricular  education  is  typically 
designed  to  develop  self-efficacy  through  mastery  experiences  (2001,  249). 

Mastery  experiences  refer  to  the  performance  of  past  activities  and  the  resulting 
observances  regarding  whether  the  performances  were  successful  or  not.  A  noteworthy 
observation  from  this  research  is  that  if  the  individual  perceives  that  he  or  she  was 
successful  in  an  activity,  then  the  individual  will  develop  a  self-perceived  sense  of 
capability  in  being  able  to  perfonn  the  same,  or  a  similar  activity  in  the  future  (Ponton  et 


15 


al.  2001,  249).  In  the  process  of  designing  mastery  experiences,  two  important  aspects  to 
consider  are:  what  exactly  do  we  want  students  to  master;  and  how  are  we  going  to  let 
them  know?  (Ponton  et  al.  2001,  249).  After  deciding  the  important  skills  a  practicing 
engineer  should  master,  we  must  also  tell  students  why  the  chosen  skills  are  important 
and  develop  assessment  measures  that  reinforce  the  value  of  the  desired  skill  (Ponton  et 
al.  2001,  249).  In  regards  to  curriculum  development,  Ponton  et  al.,  stress  that  the 
important  point  to  consider  is  that  a  decision  should  be  made  concerning  the  specific 
skills  (behavioral  or  cognitive)  that  are  important  to  a  practicing  engineer,  and  then  create 
mastery  experiences  that  will  enhance  a  student’s  sense  of  competency  in  performing  the 
skills  (2001,249). 

A  vicarious  experience  refers  to  the  development  of  self-efficacy  in  an  activity 
after  observing  similar  persons  engaging  successfully  in  that  same  activity  (Ponton  et  al. 
2001,  249).  Vicarious  experiences  are  used  to  augment  self-efficacy  when  students  see 
similar  others  perfonning  desired  skills.  Finally,  verbal  persuasion  refers  to  developing 
self-efficacy  when  important  referents  tell  a  person  he  or  she  has  the  capability  to 
successfully  perform  an  activity  (Ponton  et  al.  2001,  249). 

1.  Cognitive  Motivation 

Studies  have  shown  that  “self-beliefs  of  efficacy  have  a  large  role  in  the  self¬ 
regulation  of  motivation  and  that  most  of  human  motivation  is  cognitively  generated” 
(Bandura  1993,  128).  For  example,  students  who  motivate  themselves  do  so  by  forming 
beliefs  about  what  they  can  do;  they  set  personal  goals  and  then  plan  a  series  of  tasks 
aimed  to  accomplish  the  goal(s).  Three  theories  have  been  postulated  by  Ponton  et  al.  that 
describe  the  aforementioned  mechanisms  of  cognitive  motivation.  These  are  goal  theory, 
expectancy  value  theory  (EVT)  and  attribution  theory. 

Ponton  et  al.  note,  “The  basic  premise  of  goal  theory  is  that  when  people  adopt 
performance  goals  that  they  wish  to  accomplish,  motivation  is  induced  by  the 
discrepancy  between  their  current  and  future  desired  (i.e.,  goal)  states”  (2001,  248). 
Recognizing  the  discrepancy  between  present  and  future  states  is  the  source  of  motivation 
towards  reaching  the  goal.  As  Ponton  et  al.  further  explain,  “A  challenging  goal  provides 
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a  greater  discrepancy  and  as  a  result,  greater  motivation  because  of  the  anticipated  self- 
satisfaction  one  hopes  to  attain  from  achieving  the  stated  goal”  (2001,  248).  Research 
suggests  that  goals  should  be  challenging,  but  not  perceived  to  be  beyond  one’s  capability 
(Ponton  et  al.  2001,  248).  Distal  goals  that  are  considered  to  be  beyond  one’s  capability 
may  be  achieved  if  proximal  sub-goals  are  set  within  one’s  scope  of  self-efficacy  (Ponton 
et  al.  2001,  248).  As  such,  self-efficacy  plays  an  important  role  in  goal  theory  (Ponton  et 
al.  2001,248). 

Goal  proximity  influences  motivation  in  that  while  distal  goals  provide  a  general 
(but  weak)  motivation  for  performance,  proximal  sub-goals  that  provide  immediate 
indicators  of  success  and  support  the  ultimate  distal  goals,  provide  a  greater  source  of 
motivation.  Accomplishing  sub-goals  can  enhance  self-efficacy,  thereby  increasing  an 
individual’s  commitment  to  the  overall,  distal  goal.  Thus,  the  theoretical  findings  suggest 
that  distal  goals  for  students  should  be  apportioned  into  sub-goals  that  motivate  behaviors 
“leading  to  the  accomplishment  of  the  sub-goals  and  ultimately  the  distal  goal  itself’ 
(Ponton  et  al.  2001,  248). 

Expectancy  value  theory  (EVT)  describes  the  form  of  cognitive  motivation 
whereby  individuals  engage  in  “activities  to  accomplish  goals  because  of  the  perceived 
value  associated  with  their  likely  outcomes”  (Ponton  et  al.  2001,  248).  EVT  holds  that  the 
“greater  the  expectation  that  a  particular  behavior  will  lead  to  a  valued  outcome,  the 
greater  is  the  motivation  to  engage  in  that  behavior”  (Ponton  et  al.  2001,  248).  EVT  also 
holds  that  the  presence  of  self-efficacy  facilitates  the  process  (Ponton  et  al.  2001,  248). 
Ponton  et  al.  suggest  that  an  important  mediating  factor  self-efficacy  includes  in  relation 
to  an  individual’s  accomplishments  is  perseverance  (2001,  248).  For  instance,  their 
research  provides  an  example  whereby  even  if  a  student  perceives  that  he  or  she  has  the 
required  capabilities  to  perfonn  engineering  coursework,  the  student  may  not  fully  delve 
into  the  “engineering  curriculum  due  to  a  feeling  of  self-inefficacy  with  regards  to 
sustaining  the  effort  required  to  complete  the  program”  (Ponton  et  al.  2001,  249). 

Attribution  theory  “describes  the  motivating  influences  of  how  an  individual 

attributes  causes  of  past  accomplishments  and  failures  thereby  affecting  future  choices  of 

behaviors”  (Ponton  et  al.  2001,  249).  According  to  attribution  theory,  if  an  individual 
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attributes  the  success  of  a  past  performance  to  hard  work  or  failure  to  a  lack  of  effort, 
then  that  same  individual  is  more  likely  to  engage  in  a  similar  perfonnance  in  the  future 
as  the  need  arises  (Ponton  et  al.  2001,  249).  As  such,  effort  attributions  are  mediated 
through  the  self-efficacy  mechanism  (Ponton  et  al.  2001,  249). 

Effort  attribution  “refers  to  how  an  individual  attributes  the  causes  of  the  effort 
expended  in  perfonning  a  chosen  behavior,  for  example,  high  effort  is  indicative  of  low 
ability”  (Ponton  et  al.  2001,  249).  Effort  attributions  are  based  on  whether  ability  is 
understood  as  being  of  “static  or  dynamic  proportions”  (Ponton  et  al.  2001,  249).  If 
ability  is  understood  to  be  static  and  temporarily  stable,  then  a  high  level  of  effort  is 
understood  as  an  indicator  of  low  self-efficacy.  Conversely,  if  ability  is  understood  to  be 
dynamic  and  capable  of  influence,  then  ability  is  enhanced  with  effort  alongside  a  parallel 
increase  in  self-efficacy  (Ponton  et  al.  2001,  249). 

Cognitive  motivation  also  works  retrospectively,  as  when  an  individual  reflects  on 
the  possible  causes  of  successes,  failures  and  effort — forming  such  evaluations  can 
influence  future  actions  (Ponton  et  al.  2001,  248).  For  instance,  an  individual  may 
interpret  through  an  activity  that  causes  stress  (either  emotional  or  physical)  that  a  lack  of 
the  essential  capability  required  for  success  is  present  (Ponton  et  al.  2001,  249).  Ponton  et 
al.,  provide  a  case  study  example  to  describe  how  physiological/emotive  arousals  can 
affect  self-efficacy.  For  instance,  when  a  student  is  engaged  in  a  stressful  assignment,  the 
student  may  interpret  induced  stressors  as  indicators  that  the  capabilities  needed  to 
successfully  complete  the  assignment  are  lacking  (Ponton  et  al.  2001,  250).  Thus, 
physiological/emotive  arousals  supply  self-efficacy  information  through  the 
interpretations  of  the  reactions  (Ponton  et  al.  2001,  249). 

Ponton  et  al.  conclude  their  research  on  the  motivational  constructs  of  self- 
efficacy  by  suggesting  that  because  of  the  important  mediating  influence  self-efficacy  has 
been  observed  to  have  on  self-motivation,  an  educator  should  understand  how  the  trait 
can  be  increased  so  that  such  motivating  principles  can  be  used  in  the  design  of  effective 
instructional  strategies  (2001,  249). 
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E.  SELF-EFFICACY  IN  COGNITIVE  DEVELOPMENT  AND 

FUNCTIONING 

According  to  Bandura,  “Conceptions  of  ability  affect  thought  processes  and 
performance  attainments  through  the  self-efficacy  mechanism”  (1993,  121).  Human 
functioning  is  influenced  by  the  beliefs  individuals  have  on  how  their  ability  can  evolve 
over  time  (Bandura  1993,  121).  For  instance,  individuals  who  consider  ability  as  a 
biologically  decreasing  aptitude  with  aging  are  likely  to  infer  poor  perfonnance  as 
indicative  of  declining  capability.  And,  they  do  little  to  utilize  their  capabilities.  Whereas, 
individuals  that  perceive  ability  as  a  skill  that  can  be  developed  and  practiced,  accomplish 
higher  achievements  (Bandura  1993,  121). 

At  this  point  in  the  literature  review,  it  is  noteworthy  to  provide  an  overview  on 
theoretical  findings  of  self-efficacy  and  cognitive  development  and  functioning.  One  such 
finding  is  that  perceived  self-efficacy  exerts  its  influence  through  four  major  processes: 
cognitive,  motivational,  affective  and  selection  (Bandura  1993,  1 17). 

1.  Cognitive  Processes 

Research  on  self-efficacy  has  shown  that  it  contributes  to  cognitive  development 
and  functioning  (Bandura  1993,  117).  Cognitive  processes,  similar  to  cognitive 
motivation  discussed  earlier,  require  a  strong  sense  of  self-efficacy  for  one  to  remain  task 
oriented  when  faced  with  pressing  situations  and  failures  that  have  social  repercussions 
(Bandura  1993,  120).  For  instance,  individuals  who  perfonn  poorly  on  tasks  may  do  so 
because  they  lack  the  requisite  skills,  or  have  the  requisite  skills,  but  lack  the  sense  of 
efficacy  to  use  the  skills  effectively  (Bandura  1993,  118).  Relative  to  this  theory  is  the 
conception  of  ability — a  belief  system  whereby  how  people  regard  ability  affects  their 
cognitive  functioning.  For  instance,  some  children  consider  ability  as  an  acquirable  skill 
that  can  be  increased  by  gaining  knowledge  and  competencies.  This  cognitive  process 
leads  to  a  functional-learning  goal,  in  which  they  seek  challenges  that  provide 
opportunities  to  expand  their  knowledge  and  competencies  and  consider  errors  as 
opportunities  to  leam  from  their  mistakes.  Other  children  see  ability  as  an  inherent 
capacity.  For  them,  perfonnance  is  diagnostic  of  their  inherent  intellectual  capacities.  As 


19 


Bandura  states,  “deficient  performances  carry  high  evaluative  threats  that  they  lack  basic 
intelligence.”  That  is  why,  they  prefer  tasks  that  reduce  errors  and  demonstrate 
proficiency  “at  the  expense  of  expanding  their  knowledge  and  competencies.” 
Furthermore,  this  acquirable  skill  view  allows  for  a  task-diagnostic  focus  aimed  at  honing 
one’s  competencies  and  mastering  challenges  (Bandura  1993,  120). 

In  reference  to  cognitive  processes,  self-efficacy  seems  to  have  a  particular 
mediating  effect  on  writing  skills  and  memory.  According  to  Bandura,  “Enhancement  of 
perceived  writing  efficacy  by  instruction  raises,  through  different  paths  of  influence, 
perceived  self-efficacy  for  academic  activities,  personal  standards  for  the  quality  of 
writing  considered  self-satisfying  and  academic  goals  and  attainments”  (1993,  136). 
Similarly,  higher  cognitive  effort  has  been  shown  to  produce  better  memory  performance 
(Bandura  1993,  121). 

2.  Motivational  Processes 

The  ability  to  practice  self-influence  by  personal  challenge  and  evaluate  reaction 
to  an  individual’s  achievements  provides  a  major  cognitive  tool  of  motivation.  Behavior 
is  motivated  and  guided  by  cognized  current  goals.  Research  indicates  that  precise, 
challenging  goals  improve  and  sustain  motivation  (Bandura  1993,  130). 

3.  Affective  Processes 

According  to  Bandura,  “People’s  beliefs  in  their  capabilities  affect  how  much 
stress  and  depression  they  experience  in  threatening  or  difficult  situations,  as  well  as  their 
level  of  motivation”  (1993,  132).  This  affective  processing  is  the  emotional  mediator  of 
self-efficacy  belief. 

4.  Selection  Processes 

Self-efficacy  can  influence  the  decisions  individuals  make  by  affecting  their 
selection  of  activities  and  environments.  For  instance,  “people  avoid  activities  and 
situations  they  believe  exceed  their  coping  capabilities.  Yet,  they  readily  opt  for 
challenging  activities  and  select  situations  they  judge  themselves  capable  of  handling” 
(Bandura  1993,  134).  Research  has  suggested  that  the  higher  an  individual’s  self-efficacy 
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within  a  domain,  the  more  career  options  he  or  she  envisions  as  possible  and  the  greater 
the  interest  he  or  she  demonstrates  in  the  chosen  career  (Bandura  1993,  135).  Similarly, 
high  self-efficacy  mediates  the  process  whereby  an  individual  can  better  prepare 
himself/herself  educationally  for  various  occupations  along  the  chosen  career  path 
(Bandura  1993,  135).  One  cannot  underestimate  the  lasting  impact  these  findings  suggest, 
which  is  that  high  self-efficacy  within  an  occupational  domain  can  positively  influence 
one’s  “staying  power  and  success  in  difficult  occupational  pursuits”  (Bandura  1993,  135). 

F.  SELF-EFFICACY  IN  THE  DEVELOPMENT  OF  TECHNICAL  AND 

PROFESSIONAL  SKILLS 

The  literature  review  suggests  that  there  are  three  categories  of  skills  needed  by 
leaders  to  be  successful:  conceptual,  human  and  technical  (Moore  and  Rudd  2005,  69).  A 
research  model,  similar  to  our  own,  supports  this  theory  by  way  of  results  of  a  survey 
instrument  that  was  designed  with  39  competencies.  Factor  analysis  results  of  the  39 
competency  items  revealed  nine  broad  competency  areas:  four  represented  conceptual 
skills,  three  represented  human  skills  and  two  represented  technical  skills  (Moore  and 
Rudd  2005,  69). 

Research  on  leadership  competencies  has  suggested  that  conceptual  competencies 
are  the  most  important,  followed  by  human  competencies,  and  finally  with  technical 
competencies  as  the  least  important  asset  in  leadership  KSAs  (Moore  and  Rudd  2005, 
69).  Conceptual  skills  are  perhaps  the  most  important  at  top  management  levels  where 
policy  decisions,  long  term  planning,  and  broad  scale  actions  are  required  (Moore  and 
Rudd  2005,  69).  Conceptual  skills  involve  being  able  to  see  both  what  is  going  on  within 
an  entire  organization  and  how  different  components  of  the  organization  interact  and 
depend  on  one  another  (Moore  and  Rudd  2005,  26). 

Human  competencies  are  defined  as  the  “people  skills”  or  the  ability  to  work 
effectively  as  a  group  member  and  to  build  cooperative  effort  within  the  team  being  led 
(Moore  and  Rudd  2005,  69).  It  is  noteworthy  to  mention  here  that  the  NPS’s  survey 
instrument  and  SE  competency  model  encompass  such  human  competencies  in  the  form 
of  the  “teamwork”  domain.  Moore  and  Rudd  state  that  the  presence  of  human  skills  is 
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important  throughout  all  management  levels  (2005,  69).  Ideally,  human  skills  are  shown 
in  how  a  leader  perceives  and  behaves  towards  those  around  him  or  her,  including 
superiors,  peers,  and  followers.  More  importantly,  it  must  be  demonstrated  in  every 
action  of  the  leader  (Moore  and  Rudd  2005,  69).  This  is  why  the  NPS’s  competency 
model  maps  leadership  KSAs  throughout  a  system  engineer’s  career. 

Goleman  extends  the  theoretical  framework  on  leadership  competencies  from 
purely  technical  skills  and  cognitive  abilities  to  include  that  of  emotional  intelligence  (El) 
(Moore  and  Rudd  2005,  69).  He  suggests  that  there  are  five  components  to  El:  self- 
awareness,  self-regulation,  motivation,  empathy  and  social  skill  (Moore  and  Rudd  2005, 
69).  Goleman  includes  El  as  a  set  of  leadership  skills  because  he  sees  it  as  a 
distinguishing  competence  of  senior  leaders  (Moore  and  Rudd  2005,  69).  A  noteworthy 
research  finding  is  that  Goleman  found  El  to  be  twice  as  important  as  the  other  leadership 
skills  when  applied  to  all  levels  of  jobs  within  the  organizational  hierarchy.  He  also  found 
that  El  (rather  than  conceptual  skills)  explained  90  percent  of  the  difference  between  the 
effectiveness  of  lead  performers  and  average  senior  level  leaders  (Moore  and  Rudd  2005, 
69). 

An  agricultural  study  conducted  by  Moore  and  Rudd  used  data  on  land-owning 
individuals  to  assess  the  importance  and  level  of  six  leadership  skill  areas  (2005,  68). 
Their  research  cites  the  notion  that  while  “leadership  skills  can  be  taught  and  learned, 
skilled  leaders  continue  to  be  in  short  supply”  (Moore  and  Rudd  2005,  68).  To  be 
successful  as  a  leader,  an  individual  must  understand  how  to  motivate  a  group  of 
individuals  (i.e.,  students,  or  entry  level  employees)  towards  a  common  goal  (i.e.,  the 
desired  future  state)  (Ponton  et  al.  2001,  247). 

Robbins,  Bradley  and  Spicer  (2011)  designed  a  leadership  competency 

assessment  instrument  for  healthcare  administration,  with  52  items  classified  as  technical 

skills,  industry  knowledge,  analytic  and  conceptual  reasoning  or  interpersonal  and  El 

(Moore  and  Rudd  2005,  70).  Their  findings  conclude  that  industry  knowledge  should  be 

included  as  a  domain  of  skills  due  to  the  complex  nature  of  the  health  care  industry 

(Moore  and  Rudd  2005,  70).  Industry  knowledge  is  defined  as  having  expansive 

knowledge  of  the  industry  and  the  organization.  It  is  “essential  to  creating  the  broad 

22 


outlook  needed  by  leaders  to  produce  an  organizational  vision  and  strategies  to 
accomplish  that  vision”  (Moore  and  Rudd  2005,  70).  As  such,  one  could  argue  that  this  is 
another  domain  of  competencies  within  engineering,  and  one  that  should  be  further 
studied  to  see  if  industry  knowledge  affects  self-efficacy. 

Moore  and  Rudd  suggest  that  technical  skills  are  associated  with  understanding 
and  are  the  skills,  methods,  processes,  procedures  and  techniques  required  to  complete 
specific  activities  (2005,  69).  Their  findings  suggest  that  technical  skills  are  more 
important  at  the  lower  levels  of  administration,  and  that  specifically,  as  a  leader  moves  up 
in  the  organizational  hierarchy,  he  or  she  relies  on  the  technical  skills  of  followers  more 
than  on  his  or  her  own  technical  skills  (Moore  and  Rudd  2005,  69).  Survey  research  led 
Moore  and  Rudd  to  find  that  survey  participants  perceived  technical  skills  to  be  the  least 
important  of  the  skills  set  expected  of  leaders  (2005,  75).  Their  finding  that  technical 
skills  are  perceived  as  less  important  than  other  leadership  competencies  is  consistent 
with  literature  that  reports  the  amount  of  technical  skills  required  by  leaders  decreases  the 
higher  in  the  organizational  hierarchy  the  leaders  go,  because  of  the  tendency  to  delegate 
technical  tasks  to  lower  level  staff  (Moore  and  Rudd  2005,  69).  Finally,  Moore  and  Rudd 
recommend  that  competencies  within  the  communication  skill  area  be  included  in 
leadership  and  professional  development  programs  (2005,  70). 

G.  CHAPTER  SUMMARY 

The  literature,  thus,  suggests  that  domain  self-efficacy  aids  in  the  development  of 
an  individual’s  initial  interest  in  a  corresponding  career  area,  influences  his/her 
motivation  to  pursue  a  career  within  the  field,  as  well  as  the  behavior  towards  the 
selection  of  relevant  career  paths,  and  then  supports  the  higher  levels  of  both  academic 
performance  and  persistence  in  the  field.  Pertinent  to  NPS’s  research  on  self-efficacy 
within  engineering  is  foundational  research  perfonned  by  Lucas  et  al.  via  a  survey  model, 
which  found  that  the  role  of  self-efficacy  continues  to  be  important  past  education  and 
training,  when  individuals  have  entered  their  careers,  as  the  sociopsychological  construct 
can  be  used  to  predict  better  work  performance.  Similar  research  by  Joshi  et  al.  provides 
another  example  of  a  successful  attempt  to  measure  self-efficacy  as  a  latent  psychological 
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trait  using  a  survey  instrument.  Elaborating  on  these  and  other  research  findings  that 
suggest  that  high  levels  of  self-efficacy  are  indicative  of  the  development  of  mature 
cognitive  and  affective  processes  (which  can  be  used  to  predict  proficient  work 
performance),  NPS  has  undergone  research  to  identify,  via  a  competency  model  and  the 
SE  Competency  Survey,  relevant  engineering  KSAs  and  the  developmental  stages  of  such 
cognitive  and  affective  processes.  In  closing,  NPS  stipulates  that  self-efficacy  is  a  useful 
measure  of  academic  and  work  performance,  and  provides  research  to  support  the 
assertion  on  the  usefulness  of  the  construct  as  a  reliable  indicator  of  career  performance. 
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III.  MODEL  BASED  RESEARCH  ON  TECHNICAL  SKILLS, 
NON-TECHNICAL  SKILLS  AND  SELF-EFFICACY 
AS  REGULATORS  OF  COGNITIVE  AND 
AFFECTIVE  PROCESSES 

A.  INTRODUCTION:  COMPETENCY  MODEL  DEVELOPMENT 

Relative  to  NPS’s  research  was  the  role  of  competency  modeling,  defined  as  the 
“activity  of  determining  the  specific  competencies  that  are  characteristic  of  high 
performance  and  success  in  a  given  job”  (LaRocca  2013).  To  understand  the  relationship 
competency  modeling  has  with  the  identification  of  the  KSAs  required  of  engineers,  NPS 
developed  a  systems  engineering  career  development  competency  model.  Referred  to  in 
research  and  literature  as  the  competency  of  the  profession  of  systems  engineering 
(COMPOSE),  the  model  highlights  an  aggregate  of  the  core  technical,  technical 
management  and  professional  (i.e.,  non-technical)  KSAs  researched  from  the  current 
hiring  manuals  of  various  naval  and  engineering  enterprises  (White  2014,  4).  The  NPS 
team  feels  that  systems  engineers  need  the  professional  competencies  at  higher 
proficiency  levels  in  the  earlier  stages  of  a  career  than  disciplinary  engineers  (Whitcomb, 
Khan  and  White  2013). 

NPS  believes  the  competency  model  can  be  used  to  identify  and  evaluate  KSAs 
required  to  develop  self-efficacy  across  various  engineering  domains.  As  such,  it  can  be 
used  by  universities  to  ensure  training  and  education  meet  the  needs  of  industry.  Two  key 
objectives  of  the  COMPOSE  model  were  to:  develop  an  approach  and  methodology  to 
obtain  baseline  infonnation  needed  for  SE  competency  model  development  (for  example 
the  related  KSAs,  relevant  education,  training,  and  experience  required)  and  to  define 
career  paths  for  systems  engineers  Gobs,  assignments,  and  timing). 

Research  on  self-efficacy  has  suggested  that  an  individual’s  level  of  self-efficacy 
is  a  motivational  predictor  of  how  well  a  person  will  perform  at  almost  any  endeavor,  and 
can  be  utilized  and  developed  to  leverage  perfonnance  enhancing  advantages  (National 
Security  Leadership  Foundation  n.d.).  Thus,  it  can  be  understood  that  self-efficacy  serves 
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as  a  regulator  across  a  broad  range  of  functioning,  as  the  key  detenninant  for  human 
cognitive/affective,  motivational  and  decision-making  processes  (National  Security 
Leadership  Foundation  n.d.). 

B.  DEFINING  SE  COMPETENCIES:  CLASSIFICATION  BASED  ON 

AFFINITY 

The  NPS  research  team  developed  a  holistic  approach  to  the  design  and  structure 
of  COMPOSE  as  a  systems  engineering  competency  model.  Several  competency  models 
were  used  to  verify  the  COMPOSE  model  in  an  effort  to  combine  skills  from  different 
sources  to  generate  a  complete  scope  of  SE  KSA.  Collectively,  the  competency  models 
were  mapped  to  the  41  competencies  of  the  DAU  SE  Competency  Model  to  fonn  the 
basis  of  the  competency  list  for  the  COMPOSE  model.  The  competency  classification 
verification  process,  based  on  affinity,  was  achieved  by  mapping  KSA  to  proficiencies 
within  the  competencies  using  key  words  from  Bloom’s  taxonomy,  and  then  correlating 
the  KSA  to  a  specific  cognitive  or  affective  category.  In  the  current  version  of  the 
COMPOSE  model  there  are  27  technical  and  technical  management  competencies,  14 
professional  competencies,  and  2,914  KSAs  along  three  notional  skill  levels,  discussed  in 
the  next  section  (Whitcomb,  Khan  and  White  2013). 

C.  COGNITIVE  AND  AFFECTIVE  LEARNING  DOMAINS:  BLOOM’S 

TAXONOMY 

Of  noteworthy  significance  to  the  purpose  of  this  thesis  is  the  research  conducted 
by  White  on  the  development  of  the  COMPOSE  model,  and  its  identification  and 
evaluation  of  the  cognitive  and  affective  domains  associated  with  SE  KSA  (2014).  The 
COMPOSE  model  categorizes  KSAs  in  the  form  of  Bloom’s  taxonomy  into  either  the 
cognitive  or  affective  domain,  for  each  KSA  in  both  technical  and  non-technical  SE 
competencies.  As  such,  the  COMPOSE  model  allows  for  investigation  into  the  break  out 
into  cognitive  and  affective  processes  (reference  Figures  1-3).  As  White’s  research 
(reference  Figures  2  and  3)  reflects,  67  percent  of  the  COMPOSE  model’s  KSAs  are 
mapped  into  the  cognitive  domain,  with  the  remaining  33  percent  being  in  the  affective 
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domain  (2014,  18).  As  reflected  in  Figures  1-3,  the  COMPOSE  model  illustrates  how  the 
specific  SE  related  competencies  that  are  characteristic  of  high  performance  and  success 
in  SE  related  jobs  can  be  apportioned.  With  33  percent  (Figure  1)  of  the  KSAs  being 
grouped  in  the  affective  domain,  this  illustrates  that  this  is  important  to  the  development 
of  a  SE.  In  addition,  the  NPS  research  provides  weight  to  current  research  findings  that 
suggest  that  affective  learning  domains,  such  as  to  value  and  respond  (Figure  3),  are 
“important  constructs  today’s  students  must  possess”  and  are  essentially  a  deciding  factor 
to  the  successful  career  achievement  of  lead  performers  (Joshi  et  ah,  2012;  Moore  and 
Rudd  2005). 

Furthermore,  the  analysis  illustrates  how  cognitive  and  affective  levels  span  the 
entire  domain  of  SE  (Figures  2  and  3).  In  COMPOSE,  every  level  within  each  learning 
domain  is  represented,  so  as  such,  it  is  important  to  include  all  of  the  levels  in  aspects  of 
assessing  self-efficacy,  as  NPS  does  so  via  the  SE  Competency  Survey.  If  one  can  identify 
his/her  level  of  self-efficacy  in  each  of  the  SE  domains,  and  by  doing  so,  also  quantify 
KSA  associated  with  the  cognitive/affective  learning  domains,  it  is  reasonable  to  assume 
one  can  then  also  gain  an  understanding  of  where  he/she  is  at  in  being  able  to  perform 
domain  specific  tasks.  This  is  significant  as  research  indicates  an  individual’s  perceived 
level  of  self-efficacy  greatly  influences  his/her  ability  to  achieve  domain  specific  tasks 
(Lucas  and  Cooper  2004,  12). 
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COMPOSE  Cognitive  and  Affective 
Breakout 


■  Cognitive 

■  Affective 


Figure  1.  COMPOSE  Model’s  Cognitive  and  Affective  Category  Breakout 

(from  White  2014,  54). 
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■  Understand  (U) 

■  Apply  (AP) 

■  Analyze  (AN) 

■  Evaluate  (EV) 

■  Create  (C ) 


Figure  2.  Bloom’s  Cognitive  Levels  within  the  COMPOSE  Overall  Model 

(from  White  2014,  55). 
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Figure  3.  Bloom’s  Affective  Levels  within  the  COMPOSE  Overall  Model 

(from  White  2014,  56). 


In  reference  to  the  COMPOSE  model,  the  cognitive  domain  includes  knowledge, 
critical  thinking  and  the  development  of  intellectual  skills  (White  2014,  15),  whereas,  the 
affective  domain  deals  with  the  emotions,  feelings  and  attitudes  of  systems  engineers. 
Affective  KSAs  describe  growth  in  awareness,  attitude,  emotion,  changes  in  interest, 
judgment  and  the  development  of  appreciation  (White  2014,  16).  While  most  educators 
do  not  include  the  affective  learning  domain  in  the  development  of  their  curriculum, 
White  states,  “these  outcomes  are  especially  critical  to  the  success  of  systems  engineers 
as  [they]  lead  systems  projects,  negotiate  outcomes  with  a  diverse  group  of  stakeholders, 
make  value  judgments  and  must  have  the  ability  to  deliberately  take  the  systems 
perspective”  (2014,  16). 

The  development  of  cognitive  and  affective  domains  is  categorical  and 
hierarchical,  with  increasing  degrees  of  complexity  (Clark  1999).  Bloom’s  theory  of 
categorical  and  hierarchical  domains  of  learning  is  mirrored  in  the  COMPOSE  model,  as 
it  utilizes  Bloom’s  taxonomic  approach  to  aid  in  the  description  and  classification  of 
KSAs  required  of  engineers. 


29 


To  illustrate,  as  an  engineer  progresses  from  various  levels  of  experience,  it  is 
assumed  he/she  improves  in  knowledge,  skills  and  abilities  related  to  the  field.  This  is 
significant  to  mention  as  research  suggests  that  skill  development  is  a  continuous  process, 
and  as  knowledge  and  procedures  continue  to  be  compiled,  more  “elegant  task  strategies 
emerge”  (Kraiger,  Ford  and  Salas  1993,  315). 

The  NPS  competency  model  takes  the  taxonomic  approach  one  step  further  by 
providing  an  applicable  method  to  document  and  assess  at  which  points,  and  at  which 
depth,  should  an  engineer  demonstrate  particular  KSAs  within  the  27  core  technical  and 
14  non-technical  competencies  identified. 

D.  SELF-EFFICACY  AND  KSAS:  PREDICTORS  OF  WORK 

PERFORMANCE 

Research  on  self-efficacy  within  engineering  suggests  that  while  work  experience 
will  facilitate  the  acquisition  of  considerable  knowledge  and  the  learning  of  new  skills,  it 
also  has  the  advantage  of  enabling  engineers  to  carry  out  work  tasks  where  their 
performance  will  be  assessed  both  by  workplace  professionals  and  the  engineers 
themselves  (Lucas  et  al.  2009,  741). 

As  previously  described,  the  41  competencies  included  in  the  COMPOSE  model 
can  be  viewed  as  the  core  technical  and  non-technical  competencies  vital  to  the 
performance  of  an  engineer.  Select  KSAs  represented  in  the  Systems  Engineering 
Competency  Survey  allow  a  means  whereby  competencies  defined  by  the  COMPOSE 
model  can  be  applied  in  a  practical  means  by  workplace  professionals  and  engineers  to 
assess  performance.  As  such,  the  survey  instrument  aids  in  the  assessment  a  person 
within  a  supervisory  role  can  make  of  where  an  engineer  is  at  in  his/her  respective  field  in 
tenns  of  perfonnance,  and  it  also  offers  a  means  by  which  an  engineer  can  measure 
his/her  own  level  of  self-efficacy  in  perfonning  engineering  functions.  This  is  significant 
as  research  linking  self-efficacy  to  perfonnance  indicates  it  “continues  to  be  important 
once  individuals  have  entered  their  careers,  predicting  better  work  performance”  (Lucas 
et  al.  2009,  740). 
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A  survey  of  114  research  reports  on  self-efficacy  and  work-related  performance 
show  self-efficacy  reliably  “predicts  higher  levels  of  work  perfonnance  across  a  wide 
variety  of  types  of  organizations  and  activities"  (Stajkovic  and  Luthans  1998).  For 
instance,  in  business  it  has  been  observed  that  real-estate  agents  with  higher  self-efficacy 
in  their  competencies  have  higher  sales,  and  otherwise  outperform  peers  with  lower  self- 
efficacy.  Similarly,  “causal  inferences  about  the  effect  of  self-efficacy  on  work 
performance  is  supported  by  experimental  research  that  uses  random  group  assignment 
and  the  manipulation  of  self-efficacy”  (Lucas  et  al.  2009,  740).  Research  also  suggests 
that  self-efficacy  links  with  cognitive/affective  behaviors  such  as  innovation  and 
opportunity  recognition  (Ardichvili,  Cardozo  and  Sourav  2003). 

1.  Role  of  Cognitive  and  Affective  Competencies  in  Job  Performance 

Research  suggests  that  an  individual’s  level  of  self-efficacy  is  a  motivational 
predictor  of  how  well  a  person  will  perform  at  almost  any  endeavor,  and  can  be  utilized 
and  developed  to  leverage  its  performance  enhancing  advantages  (National  Security 
Leadership  Foundation  n.d.).  An  individual’s  belief  in  their  level  of  self-efficacy  (or 
perceived  self-efficacy)  greatly  influences  their  ability  to  achieve  domain  specific  tasks. 
In  turn,  self-efficacy  serves  as  a  regulator  across  a  broad  range  of  functioning,  as  the  key 
determinant  for  human  cognitive,  affective,  motivational  and  decision-making  processes 
(National  Security  Leadership  Foundation  n.d.). 

Research  on  the  role  of  cognitive  learning  strategies  suggests  that  individual 
differences  exist  in  the  extent  to  which  knowledge  can  be  accessed  or  applied  more 
rapidly  or  fluidly  (Kraiger,  Ford  and  Salas  1993,  311).  One  theory  relating  the  role  of 
cognitive  learning  strategies  suggests  that  through  continued  practice,  complex  behaviors 
are  internalized.  The  greater  the  internalization,  the  more  cognitive  resources  are 
available  for  higher  level  functions  or  strategy  development  (Kanfer  and  Ackerman 
1989).  In  context  to  how  strategy  development  relates  to  the  NPS  SE  competency  model 
and  survey  instrument,  it  is  the  “broad  range  of  mental  activities  [i.e.,  KSAs]  that 
facilitate  knowledge  acquisition  and  application”  (Prawat  1989).  Thus,  as  knowledge  is 
acquired,  it  facilitates  the  application,  or  performance  of  work. 
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E.  CONCLUSION 

The  analysis  aimed  to  look  at  the  cognitive  and  affective  learning  processes  of 
each  KSA  for  the  professional  development  SE  competencies  identified  through  the 
analysis  of  various  engineering  development  models.  Skills  are  a  composite  of  abilities, 
techniques,  and  knowledge.  They  are  developmental  and  incremental,  and  reference- 
based  for  a  desired  application,  in  our  case,  systems  engineering  (Clark  1999).  The 
COMPOSE  model,  which  serves  as  a  guide  to  identify  and  evaluate  the  KSAs  required  to 
be  a  proficient  engineer,  identifies  the  technical  and  non-technical  competencies  across 
three  notional  proficiency  levels  within  each  competency.  Furthermore,  the  COMPOSE 
model  includes  14  non-technical  competencies  in  the  framework  of  professional  skills  a 
leader  (especially  one  within  an  engineering  discipline)  should  develop,  and  offers  an 
analysis  of  how  the  competencies  can  be  decomposed  into  individual  KSAs.  The  thesis 
argues  that  self-efficacy,  through  competency  modeling  and  surveying,  can  be  enhanced 
within  an  individual,  and,  as  a  predictor  of  work  performance,  can  aid  in  the  development 
of  proficient,  highly  skilled  engineers. 

F.  CHAPTER  SUMMARY 

The  chapter  focused  on  describing  NPS’s  efforts  on  applying  model  based 
research  of  technical  and  non-technical  skills  and  self-efficacy  to  develop  a  competency 
model.  The  chapter  began  with  a  description  of  the  process  whereby  NPS  developed  a 
systems  engineering  career  development  competency  mode — COMPOSE.  The  chapter 
also  detailed  how  the  COMPOSE  model  maps  relevant  cognitive  and  affective  processes 
to  SE  specific  KSAs.  The  NPS  research  team  contends  that  the  competency  model  can  be 
used  to  identify  and  evaluate  KSAs  required  to  develop  self-efficacy  across  various 
engineering  domains.  And,  as  such,  the  model  can  be  used  by  universities  to  ensure 
education  and  training  meet  the  needs  of  industry.  The  chapter  concluded  with  the 
argument  that  self-efficacy,  through  competency  modeling  as  the  basis  for  surveying,  can 
aid  in  the  development  of  proficient,  highly  skilled  engineers. 
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IV.  MEASURING  SELF-EFFICACY:  DEVELOPMENT  OF  A 
SELF-EFFICACY  MEASUREMENT  INSTRUMENT  AND 
ANALYSIS  OF  NPS  STUDY 


A.  HOW  DO  SELF-EFFICACY  ITEMS  IN  THE  SURVEY  INSTRUMENT 

RELATE  TO  THE  COMPOSE  MODEL? 

In  the  previous  chapter,  the  implications  for  possible  applications  of  the 
COMPOSE  model  as  a  source  of  self-efficacy  related  KSAs  was  discussed.  This  chapter 
is  an  extension  of  the  research  previously  discussed,  and  aims  to  demonstrate  how  the 
survey  instrument  developed  by  the  SE  department  at  NPS  can  be  used  to  1)  aid  in  the 
development  of  competent  systems  engineers  and  2)  be  used  as  a  tool  to  predict  career 
performance.  The  Systems  Engineering  Competency  Survey  is  a  survey  instrument 
initially  designed  with  a  composite  of  questions  aimed  to  measure  the  level  of  self- 
efficacy  perceived  by  entry  level  systems  engineers.  The  self-efficacy  items  in  the  The 
Systems  Engineering  Competency  Survey  relate  to  the  COMPOSE  model  in  that  they 
pose  the  cognitive  and  affective  KSAs  measures  as  questions.  In  other  words,  the  survey 
items  relay  the  KSAs  exhibited  in  the  model  as  measures  of  performance. 

B.  NPS  SURVEY  INSTRUMENT:  BACKGROUND  AND  METHODOLOGY 

The  primary  method  utilized  to  measure  self-efficacy  in  systems  engineering 
students  was  via  the  Systems  Engineering  Competency  Survey.  The  survey  instrument  had 
65  items  meant  to  measure  the  knowledge,  skills  and  abilities  that  are  directly  related  to 
meeting  educational  objectives  one  would  expect  of  a  master’s  systems  engineering 
program.  The  survey  items  were  developed  by  subject  matter  experts,  using  information 
collected  during  the  development  of  a  DOD  competency  model  for  the  domain  of 
systems  engineering,  and  as  such  are  believed  to  have  face  validity. 

The  survey  items  were  written  to  measure  self-confidence  in  performing  systems 
engineering  related  tasks  that  included  understanding  the  limits  of  systems  engineering 
technology  and  techniques  (critical  thinking),  translating  user  requirements  into  technical 
requirements  (systems  engineering),  leading  a  team  of  technical  professionals 

(teamwork),  and  being  able  to  track  and  monitor  the  progress  of  a  project  (project 
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management).  The  Systems  Engineering  Competency  Survey  asked  students  to  rate  their 
level  of  self-efficacy  on  a  linear  scale  of  0  to  100,  with  0  representing  “no  confidence” 
and  100  being  ‘complete  confidence’  in  being  able  to  perform  the  skills.  The  expectation 
was  that  the  survey  items  constituted  tasks  that  graduates  would  be  able  to  perform  if 
they  were  to  be  successful  in  systems  engineering  work. 

The  survey  was  delivered  in  a  pre-  and  post-mode  to  two  consecutive  SE  cohorts 
(categorized  as  RT19  and  RT19A,  respectively).  The  RT19  pre-rendition  of  the  SE 
Competency  Survey  was  deployed  to  an  SE  cohort  in  2011  during  the  middle  of  the 
students’  first  quarter.  The  RT19A  cohort  was  given  the  pre-survey  at  the  beginning  of 
their  program,  in  2012.  Then,  each  of  the  RT19  and  RT19A  post-surveys  were 
administered  towards  the  end  of  the  each  of  the  cohorts’  program. 

1.  SE  Competency  Survey:  Questionnaire  Analysis  Using  Factor 
Analysis  and  Cronbach’s  Alpha 

Following  an  example  set  by  Lucas  et  ah,  NPS  used  the  statistical  technique  of 
factor  analysis  (FA)  to  test  for  underlying  dimensions  of  the  self-efficacy  items 
embedded  in  the  survey  (2009,  744).  As  defined  by  Fricker,  Kulzy  and  Appleget: 

factor  analysis  is  a  method  for  identifying  latent  traits  from  question-level 
survey  data.  It  is  useful  in  survey  analysis  whenever  the  phenomenon  of 
interest  is  complex  and  not  directly  measurable  from  a  single  question.  In 
such  situations,  it  is  necessary  to  ask  a  series  of  questions  about  the 
phenomenon  and  then  appropriately  combine  the  resulting  responses  into  a 
single  measure  or  factor.  Such  factors,  then,  become  the  observed 
measures  of  the  unobservable  or  latent  phenomenon.  (2012,  1) 

If  the  FA  results  indicate  that  an  established  scientific  reliability  goal  is  achieved  for  each 
survey  item,  “the  latent  variables  will  have  useful  and  interpretable  meanings  that  provide 
additional  insight  into  the  characteristics  of  the  populations  being  studied”  (Fricker, 
Kulzy  and  Appleget  2012,  2). 

For  the  purposes  of  our  study,  the  latent  phenomenon  was  defined  to  be  the 
cognitive  trait(s)  of  self-efficacy  to  perform  each  competency.  We  used  the  technique  of 
FA  as  a  method  to  extract  and  label  each  latent  competency,  which  we  then  grouped  into 
one  of  the  four  systems  engineering  domains.  And,  again  following  an  example  set  forth 
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by  Lucas  et  al.,  the  criteria  for  establishing  the  scientific  reliability  goal  was  that  the  FA 
result  for  each  survey  item  must  be  .6  or  higher  on  its  “primary  component,”  and  less 
than  .4  on  any  other  component.  We  define  primary  component  as  the  domain  we 
intended  the  competency  to  fall  under.  So,  for  instance  we  intended  for  each  competency 
to  fall  under  project  management,  teamwork,  systems  engineering  or  critical  thinking. 
Factors  were  grouped  together  based  on  intended  domain  and  the  FA  value  obtained. 

The  purpose  of  the  factor  analysis  was  to  detennine  which  items  of  the  Systems 
Engineering  Competency  Survey,  if  any,  could  be  considered  to  be  reliable  indicators  of 
self-efficacy,  within  one  of  the  four  specific  domains  of  systems  engineering.  The  factor 
analyses  were  run  twice,  once  for  each  pre-survey  rendition  of  the  SE  Competency 
Survey.  The  method  of  FA  allowed  the  NPS  research  team  to  delineate  and  group  the 
competencies,  based  on  the  value  assigned  to  each  competency  from  the  survey  results. 
As  stated  above,  competencies  that  loaded  with  a  value  of  .6  or  higher  in  the  primary 
component  (domain)  were  retained  for  further  analysis.  Competencies  less  than  .4  were 
excluded  from  the  analysis  to  strengthen  the  identification  of  the  four  specific  self- 
efficacy  components. 

Cronbach’s  alpha,  a  statistical  measure  used  to  test  the  validity  and  reliability  of 
survey  data,  was  then  run  for  each  of  the  65  RT19  and  RT19A  survey  items,  and  again  for 
each  survey  set  as  a  whole.  Cronbach’s  alpha  was  observed  to  be  .989  and  .985  for  the 
RT19  and  RT19A  surveys,  respectively — suggesting  that  overall,  the  survey  items  have  a 
relatively  high  internal  consistency. 

2.  Rotated  Factor  Analysis 

The  responses  of  each  pre -rendition  of  the  RT19  and  RT19A  survey  were  used  in 
the  factor  analysis  to  test  for  underlying  dimensions  within  the  self-efficacy  survey  items. 
In  FA,  the  relationship  of  the  factors  to  each  other  is  determined  by  a  rotation  technique 
selected  during  the  analysis.  Following  an  example  set  forth  by  Lucas  et  al.,  we  chose  to 
rotate  our  FA  results  using  the  statistical  rotation  technique  referred  to  as  Varimax.  In 
FA,  the  rotated  component  matrix  displays  the  loadings,  or  value  groupings,  of  variables 
on  factors.  Variables  are  grouped  based  on  “high  loadings”  (.6  or  higher)  and  similar 
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values  across  components.  If  the  factor  axes  are  rotated,  the  loading  of  a  variable  on  one 
factor  is  maximized  while  its  loading  on  the  other  factors  is  minimized,  thereby  making 
the  factor  structure  easier  to  interpret  (Daniel  2013).  For  the  context  of  this  paper,  we 
define  the  phrase  “loaded  on”  or  “high  loadings”  to  refer  to  the  way  in  which  FA  results 
are  displayed.  Survey  items  with  a  value  of  .6  or  higher  are  said  to  “load”  strongly  - 
meaning  they  can  be  assumed  to  reliably  measure  the  latent  construct  one  is  examining. 
FA  results  display  survey  items  within  matrix  columns.  These  columns  are  suggested  as 
being  domains. 

The  significance  of  using  the  Varimax  rotation  method  after  performing  FA  is  that 
the  Varimax  rotation  attempts  to  simplify  the  columns  of  the  factor  matrix,  achieving  the 
maximum  simplification  of  component  identification.  This  also  results  in  factors  that  are 
independent  of  each  other  (Daniel  2013).  For  the  purposes  of  our  study,  this  was 
important  to  do  as  we  aimed  to  1)  test  the  hypothesis  that  self-efficacy  can  be  measured 
and  improved  upon  by  a  rigorous  education  and  training  and  2)  identify  reliable  measures 
of  self-efficacy  in  each  of  the  four  systems  engineering  domains. 

When  the  responses  were  subjected  to  rotated  factor  analysis,  11  components 
emerged  both  times,  with  the  four  expected  domains  of  systems  engineering  (i.e.,  critical 
thinking,  systems  engineering,  teamwork  or  project  management)  emerging  as  major 
components.  Again,  the  component  selection  was  detennined  using  competencies  that 
loaded  on  a  .6  or  higher  within  the  intended  systems  engineering  domain  and  excluding 
the  competencies  that  loaded  strongly  (.6  or  higher)  on  more  than  one  domain,  or  scored 
a  value  lower  than  .4.  Student  data  for  each  cohort  set  was  then  subjected  to  two  sample 
t-tests,  to  examine  if  any  statistically  significant  difference  was  observed  between  the 
mean  pre-  and  post-self-efficacy  scores. 

C.  GENERAL  RESULTS 

Using  the  factor  analysis  results  of  the  survey  items  tested  against  each  other,  it 
was  observed  that  some  survey  items  loaded  on  more  than  one  component — thereby 
indicating  that  the  items  may  need  to  either  be  re-evaluated  (so  as  to  measure  one  specific 
domain)  or  be  eliminated  from  future  renditions  of  the  survey.  A  possible  follow  on 
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procedure,  now  that  factor  analysis  has  been  perfonned  twice,  could  be  a  cross  analysis 
of  each  of  the  SE  Competency  Survey  factor  analysis  results  to  determine  which,  if  any, 
of  the  survey  items  can  be  used  for  further  survey  analysis.  Taking  into  consideration 
each  of  the  factor  analysis  results,  we  may  re-combine  certain  survey  items  or  re-write  a 
survey  to  measure  self-efficacy  in  the  SE  domains  of  teamwork,  project  management, 
systems  engineering  and  critical  thinking. 

It  is  noteworthy  to  mention  that  the  data  reflected  in  Tables  1  and  2  are  a  subset  of 
the  65  survey  items  that  were  originally  presented  to  the  students  in  the  pre-  and  post¬ 
surveys.  The  survey  items  represented  are  those  that,  through  factor  analysis,  were 
observed  to  load  on  a  single,  distinct  SE  domain. 


Table  1 .  Pre  to  Post  Education  Change  in  Levels  of  Self-Efficacy,  AY  2011 

Cohort 


A.  Self-efficacv:  Current  skill  levels  in 

Teamwork 

Percent  a' 
all  res] 

vg.  across 

rnnses 

Response  changes  from 
pre-  to  post-event  survey 
and  sign  test 

Response  rate  at  post-test,  17/17=100% 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

Help  two  team  members  resolve  serious 
personal  differences  so  they  can  work 
effectively  with  the  team. 

74 

79 

17 

5 

-0.955 

Help  make  sure  your  teammates  deliver 
on  the  schedule  that  they  agreed  to 
support. 

76 

78 

17 

2 

-0.457 

Talk  with  your  team  members  for  the 
purpose  of  assessing  their  interests  and 
skills  in  order  to  ensure  needed  team 
capabilities  are  included. 

75 

79 

16 

4 

-0.716 

Identify  important  personal  relationships 
that  must  be  considered  to  complete  a 
project. 

69 

74 

14 

5 

-0.920 

Determine  an  operating  plan  once  all  team 
members  understand  their  interests  and 
skills. 

66 

75 

14 

9 

-1.9316* 

Get  your  team  members  at  project  start  to 
make  personal  commitments  to  their 
responsibilities. 

69 

70 

14 

1 

-0.143 
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Give  constructive  criticism  to  a  poorly 
performing  team  member  that  helps  them 
improve  their  performance. 

Facilitate  a  team  to  reach  agreement  when 
there  had  been  conflicting  views  about 
what  steps  should  be  taken  next. 


B.  Self-efficacy:  Current  skill  levels  in 

Critical  Thinking 


Examine  the  statement  of  a  problem  to 
identify  its  key  elements. 

Identify  and,  if  necessary,  challenge  the 
assumptions  in  a  problem  statement. 

Examine  what  is  known  and  unknown 
about  a  problem  to  create  a  logical 
argument  for  the  best  solution. 

Read  a  policy  paper  and  identify  any 
assumption  that  is  unsupported  by  the 
evidence. 


Identify  the  critical  questions  of  fact  that 
are  missing  from  a  line  of  reasoning. 

Find  an  inconsistency  in  an  argument  the 
first  time  you  read  a  document. 

Recognize  any  logical  inconsistencies  in  a 
presentation  by  a  speaker. 

Judge  the  strengths  and  weaknesses  of  the 
evidence  supporting  your  logic. 


C.  Self-efficacy:  Current  skill  levels  in 


14  2 


-0.212 


Percent  avg.  across 
all  responses 


Response  changes  from 
pre-  to  post-event  survey 
_ and  sign  test _ 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

72 

76 

17 

4 

-1.005 

73 

71 

17 

-2 

0.527 

68 

74 

17 

6 

-1.102 

14  6 


14  4 


-0.838 


-1.020 


-0.078 


-0.824 


Percent  avg.  across 
all  responses 


Response  changes  from 
pre-  to  post-event  survey 


38 


Use  projected  warfighting  capability  gaps 
to  estimate  the  future  need  for  a  new 
system. 

60 

74 

17 

14 

-2.9528** 

Translate  the  system  capabilities  that  you 
have  co-developed  with  stakeholders  into 
technical  requirements. 

64 

71 

17 

7 

-1.133 

Determine  whether  a  set  of  system 
requirements  are  feasible. 

64 

70 

17 

6 

-1.0415 

Create  a  feasible  design  that  meets  the 
system  requirements. 

65 

67 

17 

2 

-0.4564 

Pick  a  technology  for  a  system 
considering  that  the  technology  readiness 
level  is  estimated,  and  uncertain. 

47 

68 

17 

21 

-3.8973*** 

Recognize  when  you  can  reduce  the 
complexity  of  your  completed  system 
design. 

59 

68 

17 

9 

-2.0062* 

Prioritize  the  requirements  that  a  future 
customer  has  for  a  system,  service,  or 
construct. 

68 

72 

17 

4 

-0.7142 

Design  a  system  that  fits  within  its  desired 
total  ownership  cost. 

55 

66 

17 

11 

-2.0505* 

Determine  what  evidence  is  required  as 
data  to  demonstrate  that  a  system  meets  its 
requirements. 

61 

71 

17 

10 

-1.9378 

Meet  with  20  stakeholders  with  very 
different  needs  and  determine  the  design 
that  serves  them  best. 

55 

70 

16 

15 

-2.5056 

Create  a  system  architecture  that  has 
appropriate  elements,  attributes,  and 
relationships  to  be  able  to  be  used  for 
system  development  and  decision  making. 

61 

69 

17 

8 

-1.3956 

Formulate  a  methodology  and  approach 
for  solving  a  systems  engineering  problem 
in  an  unfamiliar  subject  area. 

62 

70 

17 

8 

-1.6069 

Decompose  a  problem  into  its  solvable 
parts. 

76 

74 

17 

-2 

0.6378 

Compare  a  number  of  proposals  and 
identify  the  one  with  the  strongest  line  of 
reasoning  for  how  to  solve  the  problem. 

72 

72 

17 

0 

-0.1535 

Listen  to  20  stakeholders  with  very 
different  needs  and  define  a  set  of 
requirements  that  meet  their  needs. 

61 

69 

14 

8 

-1.6897 

39 


Create  a  rough  vision  for  a  design  solution 
in  an  hour  for  requirements  that  you  have 
just  received. 

Know  how  and  when  to  challenge  the 
assumptions  for  a  design. 

Collect  and  interpret  stakeholder  needs 
related  to  a  given  project. 

Use  your  technical  knowledge  to  make 
contributions  to  a  new  design  proposal. 
Select  the  most  appropriate  initial  design 
iteration  to  meet  a  given  set  of 
requirements. _ 

Describe  what  resources  will  be  necessary 
to  conduct  tests  to  demonstrate  that  a 
system  meets  its  requirements. 


D.  Self-efficacy:  Current  skill  levels  in 

Project  Management 


Develop  a  time-phased  budget  for  a  multi¬ 
year  project. 

Estimate  quantitatively  the  likelihood  and 
relative  impact  of  events  on  a 
development  project  that  represent  the 
most  pressing  risks. 

Create  a  work  breakdown  structure  for 
your  project. 

Identify  risk  events  that  could  have  an 
impact  on  timely  development  of  your 
system. _ 

Recognize  the  important  assumptions  that 
need  to  be  tested  at  the  start  of  a  new 
system  development  project. 

Identify  and  define  the  roles  and 
responsibilities  that  will  be  needed  to 
complete  a  project. 

Determine  what  measures  must  be  taken 
in  a  risk  mitigation  strategy  to  assess  if  the 
likelihood  or  impact  have  been  assuaged 
as  development  progresses. 


63 

71 

14 

8 

-1.3911 

63 

67 

14 

4 

-0.8112 

69 

72 

14 

3 

-0.6071 

69 

69 

14 

0 

-0.0454 

64 

71 

14 

7 

-1.2259 

58 

71 

14 

13 

-2.968** 

Percent  avg.  across 
all  responses 


Response  changes  from 
pre-  to  post-event  survey 


Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

52 

62 

16 

10 

-1.485 

59 

70 

17 

11 

-2.022* 

62 

71 

17 

9 

-1.455 

69 

71 

14 

2 

-0.449 

65 

69 

13 

4 

-1.032 

71 

75 

14 

4 

-1.032 

68 

71 

14 

3 

-0.671 

40 


Identify  in  the  development  schedule 
when  tests  must  be  completed  to 
demonstrate  that  a  system  meets  its 
requirements. 

63 

70 

14 

7 

-1.505 

Prioritize  events  in  a  risk  mitigation 
strategy  and  ideate  corrective  action. 

68 

72 

13 

4 

-0.684 

*  p<.05;  **  p<.01;  ***  p<.001.  The  t-test  was  used  to  estimate  statistical  strength  of  significance  using  the 
difference  of  means  between  the  pre-  and  post-survey  results. 


Given  a  null  hypothesis  that  any  change  found  will  be  random  and  equally  probable  in  either  direction,  the 
t-test  calculates  the  probability  that  the  number  of  changes  is  disproportionately  in  the  hypothesized 
direction  (one-tail  test). 


Table  2.  Pre  to  Post  Education  Change  in  Level  of  Self-Efficacy,  AY  2012 

Cohort 


A.  Self-efficacy:  Current  skill  levels 

in  Teamwork 

Percent  avg.  across  all 
responses 

Response  changes  from  pre-  to 
post-event  survey  and  sign  test 

Response  rate  at  post-test, 
11/11=100% 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

Ensure  team  members  follow 
agreed  upon  rules  for  meetings. 

67 

89 

11 

22 

-5.6903*** 

Help  two  team  members  resolve 
serious  personal  differences  so  they 
can  work  effectively  with  the  team. 

59 

81 

11 

22 

-4.2807*** 

Help  make  sure  your  teammates 
deliver  on  the  schedule  that  they 
agreed  to  support. 

57 

79 

11 

22 

-5.3094*** 

Talk  with  your  team  members  for 
the  purpose  of  assessing  their 
interests  and  skills  in  order  to 
ensure  needed  team  capabilities  are 
included. 

55 

76 

11 

21 

-5.0963*** 

Examine  two  design  approaches 
and  select  the  better  one  for  your 
project  to  present  as  your 
alternatives  to  your  team. 

51 

78 

11 

27 

-6.3165*** 

Identify  important  personal 

relationships  that  must  be 
considered  to  complete  a  project. 

53 

83 

11 

30 

-6.3555*** 

Help  ensure  your  team  members 
understand  the  goals,  scope,  and 
deadlines  at  the  project  start. 

55 

84 

11 

29 

-6.4187*** 

Get  your  team  members  at  project 
start  to  make  personal  commitments 

50 

76 

11 

26 

-5.4103*** 

41 


to  their  responsibilities. 

Give  constructive  criticism  to  a 
poorly  performing  team  member 
that  helps  them  improve  their 
performance. 

54 

71 

11 

17 

-2.2975* 

Facilitate  a  team  to  reach  agreement 
when  there  had  been  conflicting 
views  about  what  steps  should  be 
taken  next. 

50 

78 

11 

28 

-6.1914*** 

B.  Self-efficacy:  Current  skill  levels 

in  Critical  Thinking 

Percent  avg.  across  all 
responses 

Response  changes  from  pre-  to 
post  event  survey  and  sign  test 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

Analyze  opposing  positions  on  an 
issue  and  compare  the  quality  of 
their  evidence. 

59 

77 

11 

18 

-4  199*** 

Examine  an  engineering  problem 
and  prepare  a  written  and  oral 
statement  describing  all  essential 
elements  included  in  the  solution. 

36 

72 

11 

36 

-8  477*** 

Evaluate  data  about  an  existing 
design  problem  to  define  its  nature, 
boundaries,  and  scope. 

40 

73 

11 

33 

-8.435*** 

Find  an  inconsistency  in  an 
argument  the  first  time  you  read  a 
document. 

47 

67 

11 

20 

-4.307*** 

Recognize  any  logical 

inconsistencies  in  a  presentation  by 
a  speaker. 

48 

68 

11 

20 

-3.147** 

Identify  the  critical  questions  that 
need  to  be  answered  before  a 
feasible  design  can  be  selected. 

39 

73 

11 

34 

-6.148*** 

Judge  the  strengths  and  weaknesses 
of  the  evidence  supporting  your 
logic. 

45 

75 

11 

30 

-7  121*** 

C.  Self-efficacy:  Current  skill  levels 

in  Systems  Engineering 

Percent  avg.  across  all 
responses 

Response  changes  from  pre-  to 
post-event  survey  and  sign  test 

Pre¬ 

education 

Post¬ 

education 

N 

Percent 

increase 

T  value 

Generate  alternative  system  ideas  to 
meet  a  stakeholder  need. 

45 

76 

11 

31 

-6.955*** 

42 


Elicit  requirements  that  a  future 
stakeholder  has  for  a  system, 
service,  or  construct  using  direct 
interaction  with  them. 

39 

73 

11 

34 

-9.6184*** 

Leverage  features  or  elements  of 
existing  systems  that  can  inform  a 
new  design. 

42 

75 

11 

33 

-8.6191*** 

Translate  the  system  capabilities 
that  you  have  co-developed  with 
stakeholders  into  technical 

requirements. 

35 

70 

11 

35 

-7.7072*** 

Create  a  feasible  design  that  meets 
the  system  requirements. 

40 

72 

11 

32 

-7.6047*** 

Pick  a  technology  for  a  system 
considering  that  the  technology 
readiness  level  is  estimated,  and 
uncertain. 

33 

68 

11 

35 

-6.1698*** 

Design  a  system  that  fits  within  its 
allocated  budget,  trading  off 
schedule,  performance,  and 

lifecycle  issues. 

37 

68 

11 

31 

-4.5292*** 

Recognize  when  you  can  reduce  the 
complexity  of  your  completed 
system  design. 

39 

68 

11 

29 

-4.6898*** 

Prioritize  the  requirements  that  a 
future  customer  has  for  a  system, 
service,  or  construct. 

40 

74 

11 

34 

-6.3318*** 

Design  a  system  that  fits  within  its 
desired  total  ownership  cost. 

30 

61 

11 

31 

-4.8887*** 

Determine  what  evidence  is 
required  as  data  to  demonstrate  that 
a  system  meets  its  requirements. 

37 

66 

11 

29 

-6.8608*** 

Tailor  a  systems  engineering 
process  for  your  project. 

31 

76 

11 

45 

-9  3 137*** 

Apply  functional  analysis  during 
system  design. 

36 

75 

11 

39 

-6.8608*** 

Identify  what  tests  should  be 
conducted  to  create  data  to 
demonstrate  that  a  system  meets  its 
requirements. 

36 

68 

11 

32 

-7.4133*** 

Create  a  system  architecture  that 
has  appropriate  elements,  attributes, 
and  relationships  to  be  able  to  be 
used  for  system  development  and 
decision  making. 

29 

70 

11 

41 

-7  4799*** 

43 


Formulate  a  methodology  and 
approach  for  solving  a  systems 
engineering  problem  in  an 

unfamiliar  subject  area. 

32 

73 

11 

40 

-6.5519*** 

Respond  to  a  change  in  the 
environment  or  stakeholder  needs 
via  modification  of  the 

requirements  baseline. 

32 

61 

11 

29 

-5.2495*** 

Explain  the  ramifications  to  a 
change  in  the  environment  or 
stakeholder  needs  based  on 
modification  of  requirements. 

33 

65 

11 

32 

-5.6638*** 

Select  the  most  appropriate  initial 
design  iteration  to  meet  a  given  set 
of  requirements. 

36 

67 

11 

31 

-6.8845*** 

Describe  what  resources  will  be 
necessary  to  conduct  tests  to 
demonstrate  that  a  system  meets  its 
requirements. 

38 

68 

11 

30 

6.1492*** 

D.  Self-efficacy:  Current  skill  levels 

in  Project  Management 

Percent  avg.  across  all 
responses 

Response  changes  from  pre-  to 
post-event  survey  and  sign  test 

Recognize  the  important 

assumptions  that  need  to  be  tested 
at  the  start  of  a  new  system 
development  project. 

37 

67 

11 

31 

-4.4662*** 

Determine  what  measures  must  be 
taken  in  a  risk  mitigation  strategy  to 
assess  if  the  likelihood  or  impact 
have  been  assuaged  as  development 
progresses. 

36 

73 

11 

37 

-8.2558*** 

Prioritize  events  in  a  risk  mitigation 
strategy  and  ideate  corrective 
action. 

41 

78 

11 

37 

-8.0621*** 

*  p<.05;  **  p<.01;  ***  p<.001.  The  t-test  was  used  to  estimate  statistical  strength  of  significance  using  the 
difference  of  means  between  the  pre-  and  post-survey  results. 


Given  a  null  hypothesis  that  any  change  found  will  be  random  and  equally  probable  in  either 
direction,  the  t-test  calculates  the  probability  that  the  number  of  changes  is  disproportionately  in  the 
hypothesized  direction  (one-tail  test). 

NPS  included  only  those  items  that  were  observed  to  load  with  a  .6  or  higher  on  a 
component,  and  loaded  with  items  that  were  similarly  intended  to  measure  one  of  the 
abovementioned  domains.  And,  again,  following  the  model  set  forth  by  Lucas  et  al., 
items  that  loaded  on  more  than  one  component  were  excluded  from  the  analysis  in  order 
to  have  scales  based  on  the  items  that  capture  the  distinctness  of  the  domains  (2009,  744). 
However,  as  one  can  see,  the  two  survey  sets  yielded  different  factor  analysis  results,  and 


as  such,  reflect  different  survey  items  as  markers  of  domain  distinctness. 

44 


1.  Survey  Instrument  Analysis  Using  Rotated  Factor  Analysis  Results 

Observations  of  the  final  data  sets,  as  represented  in  Tables  1  and  2,  led  to 
examination  of  the  similarities  between  the  rotated  factor  analysis  results  to  determine 
which  survey  items  were  more  effective  in  capturing  self-efficacy  in  the  intended 
domains.  The  survey  items  to  keep,  based  on  the  rotated  factor  analysis  results  of  each 
survey,  would  be: 

Teamwork 

•  Help  two  team  members  resolve  serious  personal  differences  so  they  can 
work  effectively  with  the  team. 

•  Help  make  sure  your  teammates  deliver  on  the  schedule  that  they  agreed 
to  support. 

•  Talk  with  your  team  members  for  the  purpose  of  assessing  their  interests 
and  skills  in  order  to  ensure  needed  team  capabilities  are  included. 

•  Identify  important  personal  relationships  that  must  be  considered  to 
complete  a  project. 

•  Get  your  team  members  at  project  start  to  make  personal  commitments  to 
their  responsibilities. 

•  Give  constructive  criticism  to  a  poorly  perfonning  team  member  that  helps 
them  improve  their  performance. 

•  Facilitate  a  team  to  reach  agreement  when  there  had  been  conflicting 
views  about  what  steps  should  be  taken  next. 

Systems  Engineering 

•  Generate  alternative  system  ideas  to  meet  a  stakeholder  need. 

•  Elicit  requirements  that  a  future  stakeholder  has  for  a  system,  service,  or 
construct  using  direct  interaction  with  them. 

•  Leverage  features  or  elements  of  existing  systems  that  can  infonn  a  new 
design. 

•  Translate  the  system  capabilities  that  you  have  co-developed  with 
stakeholders  into  technical  requirements. 

•  Create  a  feasible  design  that  meets  the  system  requirements. 

•  Pick  a  technology  for  a  system  considering  that  the  technology  readiness 
level  is  estimated,  and  uncertain. 

•  Recognize  when  you  can  reduce  the  complexity  of  your  completed  system 
design. 
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•  Prioritize  the  requirements  that  a  future  customer  has  for  a  system,  service, 
or  construct. 

•  Design  a  system  that  fits  within  its  desired  total  ownership  cost. 

•  Determine  what  evidence  is  required  as  data  to  demonstrate  that  a  system 
meets  its  requirements. 

•  Create  a  system  architecture  that  has  appropriate  elements,  attributes,  and 
relationships  to  be  able  to  be  used  for  system  development  and  decision 
making. 

•  Formulate  a  methodology  and  approach  for  solving  a  systems  engineering 
problem  in  an  unfamiliar  subject  area. 

•  Select  the  most  appropriate  initial  design  iteration  to  meet  a  given  set  of 
requirements. 

•  Describe  what  resources  will  be  necessary  to  conduct  tests  to  demonstrate 
that  a  system  meets  its  requirements. 

Critical  Thinking 

•  Find  an  inconsistency  in  an  argument  the  first  time  you  read  a  document. 

•  Recognize  any  logical  inconsistencies  in  a  presentation  by  a  speaker. 
Project  Management 

•  Recognize  the  important  assumptions  that  need  to  be  tested  at  the  start  of  a 
new  system  development  project. 

•  Determine  what  measures  must  be  taken  in  a  risk  mitigation  strategy  to 
assess  if  the  likelihood  or  impact  have  been  assuaged  as  development 
progresses. 

•  Prioritize  events  in  a  risk  mitigation  strategy  and  ideate  corrective  action. 

This  subset  of  survey  items  represents  the  skills  that  were  observed  to  load  with  a 

.6  or  higher  in  a  single  component,  within  both  the  rotated  factor  analysis  tests. 

D.  RESULTS:  ANALYSIS  AND  IMPLICATIONS 

In  general,  there  is  some  evidence  to  show  that  there  was  improvement  in  the 
systems  engineering  students  as  they  progressed  through  the  program,  in  their  perceived 
level  of  self-efficacy  to  perfonn  systems  engineering  skills.  Although  not  all  items 
displayed  statistically  significant  differences  between  pre-  and  post-survey  results, 
changes  in  mean  scores  (from  pre-  to  post-)  were  generally  reported  to  be  higher, 
reflecting  an  overall  enhanced  perception  of  self-efficacy. 
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An  interesting  observation  of  the  survey  results  is  that  the  first  cohort  (RT19) 
generally  marked  their  pre-scores  to  be  much  higher  than  those  of  the  next  cohort 
(RT19A).  This  trend  leads  us  to  examine  possible  reasons  for  the  difference  in  perceived 
levels  of  self-efficacy  between  cohorts,  as  there  are  potentially  a  couple  of  reasons  for 
this  observation.  For  instance,  in  a  discussion  with  a  lead  faculty  member  who 
interviewed  the  RT19  students  about  the  scores,  it  was  deemed  that  the  students’ 
relatively  high  perception  of  their  own  ability  could  have  been  affected  by  the 
preconceived  notion  that,  prior  to  any  formal  education  or  training,  they  had  an  ability  to 
accomplish  many  of  the  competencies  because  of  the  context  of  their  military 
experiences.  This  point  is  supported  in  the  research  review  discussed  earlier,  in  which  we 
describe  the  process  by  which  mastery  experiences  (in  our  case,  that  of  the  students’ 
having  prior  experience  in  perfonning  SE  related  tasks,  albeit  through  a  military  setting), 
can  positively  affect  self-efficacy.  It  is  meaningful  to  add  to  this  discussion  that  there  was 
a  structured  difference  between  the  RT19  and  RT19A  course  format,  but  the  change  in 
structural  format  was  not  deemed  to  have  affected  the  RT19  cohort’s  pre-survey  results — 
and  in  fact  the  NPS  research  team  believes  it  would  have  impacted  the  post-survey 
results.  It  should  be  noted  that  the  RT19  pre-survey  was  deployed  during  the  middle  of 
the  students’  first  quarter.  So,  they  had  essentially  already  received  a  fair  amount  of 
educational  experience  in  fundamental  SE  concepts,  compared  to  RT19A.  The  RT19A 
cohort  was  given  their  survey  at  the  beginning  of  their  curriculum.  The  decision  of  the 
lead  faculty  to  deploy  the  RT19A  pre-survey  at  an  earlier  point  in  the  students’  master’s 
program  came  as  a  result  of  an  informal  review  process  with  students.  It  was  deemed  that 
adjusting  the  process  and  procedure  when  deploying  to  the  RT19A  cohort  would  control 
for  the  way  students  understood  the  questions.  Also,  the  project  that  was  used  for  the 
basis  of  learning  (and  by  which  the  students  would  have  been  expected  to  base  their 
experiences),  was  pedagogically  designed  as  an  authentic-learning  project. 

To  summarize,  a  general  assessment  of  the  high  pre-scores  of  the  RT19  cohort  is 
that  the  students  may  have  misjudged  (i.e.,  overestimated)  their  skills  and  abilities  in  the 
pre-test,  as  a  result  of  them  feeling  like  they  had  experience  in  performing  the  tasks  in  a 
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military  environment,  but  not  so  much  in  an  educational  environment.  Again,  we  stress 
the  role  of  mastery  experiences  in  enhancing  a  student’s  perceived  level  of  self-efficacy. 

1.  Teamwork 

Relevant  research  has  suggested  that  teamwork  provides  students  with  an 
opportunity  to  evaluate  the  skills  of  their  peers  (Ponton  et  al.  2001,  250).  When  a  student 
observes  his/her  peers  with  competencies  he/she  values,  then  the  individual  student  can 
believe  him/herself  equally  competent  in  performing  the  same  skill  (Ponton  et  al.  2001, 
250).  As  such,  our  hypothesis  was  that  the  survey  results  would  indicate  increased  levels 
of  self-efficacy  in  items  relating  to  teamwork,  as  one  would  hope  progressing  through  a 
master’s  program  at  university  would  enhance  opportunities  and  skills  necessary  for 
working  in  teams.  In  general,  results  indicate  mixed  findings  as  the  results  for  the  first 
cohort  did  not  show  significant  differences  in  the  average  response  rates  towards  many  of 
the  teamwork  aspects,  while  the  second  cohort  seems  to  have  improved  at  significant 
levels.  An  area  of  teamwork  that  seems  to  have  been  enhanced  is  the  students’  ability  to 
determine  an  operating  plan  once  all  team  members  understand  their  interests  and  skills. 
This  finding  is  indeed  interesting  as  research  has  found  teamwork  in  engineering  has  the 
advantage  of  solving  complex  problems  (Ponton  et  al.  2001, 249). 

Among  the  first  cohort’s  results,  the  skill  was  observed  to  be  significant  at  p<.01, 
with  a  percent  increase  in  pre-  and  post-results  of  nine  percent.  The  second  cohort’s 
percent  increase  between  pre-  and  post-average  was  30  percent  and  was  statistically 
significant  at  p<.001.  An  interesting  observation  one  immediately  notices  when  studying 
the  results  is  the  difference  in  the  percent  increase  of  pre-  and  post-means  between 
cohorts.  The  second  cohort  seems  to  have  indicated  lower  levels  of  self-efficacy  in  their 
pre-survey  and  so  display  a  larger  mean  difference  in  post  results.  It  is  noteworthy  to 
mention  that  the  statistical  method  of  using  a  t-test  accounts  for  the  difference  in  sample 
sizes,  and  so  the  difference  in  mean  percent  increase  across  teamwork  items  between  the 
first  and  second  cohort  provides  interesting  context  for  further  study.  It  should  be  noted 
that  a  possible  explanation  for  this  observation  is  that  lead  faculty  incorporated  course 
work  that  addressed  teamwork  into  the  curriculum  of  the  second  cohort  ( RT19A ),  but  not 
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the  first  cohort.  This  was  because,  for  the  first  cohort,  it  was  assumed  the  students  could 
effectively  work  in  teams.  This  structural  method  may  have  impacted  the  results. 

2.  Critical  Thinking 

Data  analyses  of  the  first  cohort’s  results  seem  to  indicate  that  the  self-efficacy 
scores  in  critical  thinking  did  not  go  up  enough  to  exceed  rejection  criteria  for  traditional 
hypothesis  testing,  which  would  contend  that  there  would  be  a  significant  difference 
between  pre-  and  post-averages.  All  but  two  of  the  skills  surveyed  demonstrated  higher 
averages  in  the  post-survey  results — meaning  that  even  though  the  gains  were  not 
significantly  different,  there  was  at  least  a  general  sense  of  improvement  among  students. 

An  analysis  of  the  second  cohort  yields  markedly  different  results,  with  most  of 
the  post  mean  differences  relating  to  critical  thinking  self-efficacy  skills  statistically 
significant  from  pre-survey  results  at  p<.001. 

The  mixed  results  observed  within  the  critical  thinking  items  may  be  due  to  the 
social  phenomenon  that  students  are  not  aware  of  how  well  they  can  perform  a  certain 
task  until  they  actually  have  to  utilize  the  skill  or  knowledge  set,  in  a  work  environment. 
Put  in  context,  self-efficacy  influences  the  level  of  accomplishment  realized  by  people  as 
they  perfonn  tasks  (Bandura  1997,  3). 

In  reviewing  the  critical  thinking  results,  it  should  be  noted  that  the  structural 
format  in  teaching  the  first  cohort  involved  referring  the  students  to  textbooks  on  the 
topic  of  engineering  reasoning,  whereas  the  second  cohort  was  more  formally  taught 
from  lectures  (i.e.,  through  parts  of  course  work  and  seminars)  on  what  critical  thinking  is 
and  how  it  is  used.  The  NPS  research  team  feels  that  this  change  in  structure  may  have 
impacted  the  survey  results. 

3.  Systems  Engineering 

Our  hypothesis  was  that  the  survey  results  would  indicate  increased  levels  of  self- 
efficacy  in  items  relating  to  systems  engineering,  as  it  is  expected  SE  students  will,  at  the 
least,  feel  efficient  and  capable  in  performing  standard  systems  engineering  tasks  as  they 
progress  through  a  master’s  program. 
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Contrary  to  results  observed  in  the  other  systems  engineering  domains,  the  first 
cohort’s  results  showed  significant  differences  in  several  areas.  Two  skills  in  particular 
showed  significant  differences  between  pre-  and  post-results  at  p<.01.  These  were  the 
student’s  ability  to  use  projected  warfighting  capability  gaps  to  estimate  the  future  need 
for  a  new  system  and  to  describe  what  resources  will  be  necessary  to  conduct  tests  to 
demonstrate  that  a  system  meets  its  requirements.  One  self-efficacy  item  in  particular 
reflected  statistically  significant  pre-  and  post-survey  results  at  p<.001,  which  was  the 
student’s  ability  to  pick  a  technology  for  a  system  considering  that  the  technology 
readiness  level  is  estimated,  and  uncertain. 

The  survey  offered  to  the  second  cohort  culminated  in  mean  pre-  and  post- 
differences  that  went  up  enough  to  exceed  rejection  criteria  for  traditional  hypothesis 
testing.  There  were  significant  differences  in  means  observed  in  all  self-efficacy  items 
relating  to  systems  engineering.  These  results  may  be  due  to  the  change  in  structural 
format  of  instruction  delivery  between  that  of  the  first  and  second  cohort.  The  first  cohort 
integrated  their  authentic  project  within  each  of  the  courses  in  the  curriculum.  The  second 
cohort  had  an  additional  three  consecutive  project  courses  added  to  their  curriculum  in 
which  the  instructor  reviewed  all  SE  methods  and  processes  in  the  context  of  the  project. 
This  facilitated  near  transfer  of  the  learning  of  SE  methods  and  processes  to  the  authentic 
project  context.  So,  in  theory,  key  SE  concepts  were  revisited  a  number  of  times. 
Therefore,  the  NPS  team  feels  that  this  approach  to  teaching  systems  engineering 
concepts  in  context  of  authentic -learning  projects  seems  to  improve  self-efficacy  and  is 
more  effective  in  teaching.  Also  this  finding  supports  our  assumption  that  the  systems 
engineering  students  would  perceive  higher  levels  of  self-efficacy  as  they  progressed 
through  SE  courses.  As  such,  it  can  be  assumed  that  the  education  provided  to  the 
students  positively  affected  their  ability  to  perform  systems  engineering  tasks. 

4.  Project  Management 

Similar  to  results  observed  in  the  critical  thinking  self-efficacy  items,  most  of  the 
first  cohort’s  results  did  not  go  up  enough  to  exceed  traditional  rejection  criteria  for 
hypothesis  testing.  An  exception  to  this  observation  was  in  the  skill  of  estimating 

quantitatively  the  likelihood  and  relative  impact  of  events  on  a  development  project  that 
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represent  the  most  pressing  risks,  which  had  an  increase  of  1 1  percent,  and  the  difference 
in  mean  scores  was  statistically  significant  at  p<.05. 

Results  of  the  second  cohort’s  survey  items  relating  to  project  management, 
showed  statistically  significant  differences  in  the  mean  scores  at  p<.001 — exceeding 
rejection  criteria  for  traditional  hypothesis  testing.  This  finding  supports  our  assumption 
that  systems  engineering  education  provided  to  the  students  improved  their  ability  to 
perform  tasks  related  to  project  management  skills. 

Similar  to  the  analysis  discussed  on  systems  engineering,  the  project  management 
results  may  be  due  to  the  change  in  structural  fonnat  of  instruction  delivery  between  that 
of  the  first  and  second  cohort.  As  in  the  case  in  the  systems  engineering  domain,  the 
second  cohort  was  explicitly  taught  how  SE  skills  relate  to  the  project  via  learning 
activities.  The  NPS  team  feels  that  assisting  them  in  near  transfer  of  knowledge  and 
ability  to  their  project  may  have  influenced  the  survey  results  to  reflect  an  increase  in 
self-efficacy. 

E.  CHAPTER  SUMMARY 

The  aim  of  this  research  was  to  detennine  whether  self-efficacy  in  systems 
engineering  could  be  reliably  measured  via  a  survey  instrument  among  SE  students  at  the 
Naval  Postgraduate  School. 

Overall,  the  Systems  Engineering  Competency  Survey  instrument  can  be  seen  as 
an  effective  means  to  measure  self-efficacy  in  systems  engineering  students.  As  discussed 
above,  the  survey  was  delivered  in  a  pre-  and  post-mode  to  two  consecutive  SE  cohorts 
and  was  designed  to  establish,  with  confidence,  that  the  systems  engineering  program  had 
a  measurable  influence  in  improving  the  knowledge,  skills  and  abilities  of  students  to 
perform  standard  systems  engineering  tasks.  The  results  support  the  hypothesis  that  there 
was  a  general  sense  of  improvement  in  the  systems  engineering  students  as  they 
progressed  through  the  program,  in  their  perceived  level  of  self-efficacy  to  perform 
domain  specific  systems  engineering  skills. 
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Several  format  structures  could  have  biased  the  data  and  this  should  be  strongly 
considered  when  assessing  the  results  of  the  research.  Perhaps  the  strongest  bias  was  the 
sheer  length  of  the  survey.  Offered  as  a  paper  questionnaire,  the  survey  initially  had  65 
items.  A  bias  that  may  have  occurred  due  to  the  length  of  the  first  format  was  that 
students  may  have  inadvertently  or  purposely  circled  answers  on  the  scale,  without 
considering  the  value,  in  an  effort  to  quickly  submit  the  lengthy  survey.  Despite  this,  we 
chose  to  keep  all  survey  submissions  as  there  is  no  way  to  prove  that  a  student  did  or  did 
not  feel  they  could  perform  the  skills  at  the  level  they  indicated. 

To  address  the  possible  areas  of  bias,  the  survey  has  since  been  divided  up  into 
two  surveys,  so  as  to  lessen  the  load  and  time  needed  to  take  the  survey.  We  used  the 
method  of  rotated  factor  analysis  with  the  data  obtained  from  the  first  and  second  SE 
Competency  surveys  to  observe  which  survey  items,  if  any,  loaded  on  multiple 
components,  and  took  those  survey  items  out  that  loaded  on  more  than  one  component. 
This  method  aimed  to  ensure  that  each  item  would  measure  self-efficacy  in  a  specific 
domain  of  systems  engineering,  and  aided  in  reducing  the  length  of  the  survey.  Also,  the 
two  surveys  are  now  offered  online,  and  are  paired:  systems  engineering/critical  thinking 
(‘systems  thinking’)  and  project  management/teaming.  Despite  this,  given  that  the  factor 
analysis  reports  of  each  cohort’s  survey  data  set  yielded  different  component  loading 
results  (as  unique  data  sets  were  used  as  the  basis  to  run  the  analysis),  an  important 
follow-on  step  would  be  to  merge  the  results  to  create  a  survey  instrument  that  can  more 
effectively  measure  self-efficacy  in  systems  engineering. 

Finally,  the  NPS  research  team  assessed  possible  implications  of  changes  in  the 
educational  content  and  delivery  structure  between  the  cohorts  (i.e.,  to  explicitly  go  over 
key  SE  concepts  via  projects,  course  work  and  seminars).  In  examining  the  mean 
differences  in  pre-  and  post-scores,  it  was  observed  that  statistically  significant 
differences  (particularly  within  the  systems  engineering  and  project  management 
domains),  could  have  been  due  to  this  structural  changes  in  teaching,  as  assisting  students 
in  near  transfer  of  knowledge  and  ability,  and  including  authentic-learning  projects  seems 
to  have  increased  the  perception  of  self-efficacy  in  students. 
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V.  CONCLUSIONS,  RECOMMENDATIONS,  SUMMARY  AND 
AREAS  FOR  FURTHER  RESEARCH 

A.  SUMMARY 

In  this  chapter,  we  will  discuss  conclusions,  recommendations  and  further  areas  of 
research  based  on  the  results  of  the  SE  Competency  Survey  analyses.  We  will  also 
provide  a  relevant  discussion  on  current  areas  of  research  that  tie  into  the  NPS  study  and 
can  support  our  findings  and  the  subsequent  recommendations  for  further  areas  of 
research. 

It  can  be  observed  from  the  survey  research  results  previously  discussed  that  self- 
efficacy  can  be  used  as  an  indicator  of  systems  engineering  competency  level 
achievement.  Self-efficacy,  as  it  relates  to  research  within  systems  engineering,  was 
defined  as  an  individual’s  belief  that  he  or  she  can  be  proficient  in  skills  necessary  to 
become  a  systems  engineering  professional.  System’s  engineering  self-efficacy  was  also 
defined  as  the  beliefs  in  one’s  ability  to  leam  systems  engineering  skills.  Relevant 
research  suggests  that  self-efficacy  is  influenced  by  one’s  own  sense  of  capability  and  by 
the  education,  training  and  support  received  by  others.  As  such,  self-efficacy  can  be  used 
in  systems  engineering  as  a  motivational  predictor  of  how  well  an  individual  will  perform 
tasks,  and  can  be  utilized  and  developed  to  leverage  perfonnance  enhancing  advantages 
(National  Security  Leadership  Foundation  n.d.). 

Self-efficacy  and  cognitive/affective  processes  were  shown  to  be  integral  in 
indicating  performance  in  SE  related  careers.  For  instance,  relevant  research  suggested 
that  self-efficacy  is  a  latent,  psychological  trait,  which  serves  as  a  regulator  across  a 
broad  range  of  functioning,  as  the  key  determinant  for  cognitive  and  affective  processes. 
The  NPS  team  cited  Prawat’s  (1989)  finding  that  cognitive  and  affective  skills  are 
determined  to  be  vital  in  indicating  perfonnance  in  SE  because  having  a  broad  range  of 
KSAs  enables  knowledge  acquisition  and  application.  The  NPS  team  theorized  that  the 
acquirement  of  domain  specific  knowledge  has  the  advantage  of  allowing  one  to  facilitate 
the  perfonnance  of  work. 
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A  literature  review  of  relevant  studies  on  the  topic  of  self-efficacy  led  the  NPS 
team  to  believe  that  a  gap  exists  between  being  able  to  assess  SE  KSAs  and  measure  the 
competency  of  students/entry  level  engineers  as  a  way  to  predict  future  career 
performance.  It  was  stated  that  research  by  Lucas  et  al.  (2009)  that  foundational,  as  it  is 
in  the  development  of  survey  models  to  measure  self-efficacy,  does  not  answer  whether 
or  not  self-efficacy  requires  a  higher  level  of  cognition.  Research  conducted  by  NPS, 
which  maps  KSAs  across  SE  competencies  and  categorizes  the  KSAs  into  SE  domains, 
fills  this  gap.  The  NPS  research  team  contends  that  the  research  aids  in  the  identification 
of  the  characteristic  experiences  and  knowledge  students  need  to  strengthen  self-efficacy 
in  technical  and  non-technical  fields. 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

A  conclusion  the  research  team  came  to  after  reviewing  the  results  of  the  SE 
Competency  Survey  was  that  self-efficacy  can  be  used  in  systems  engineering  to  highlight 
areas  of  potential  educational/training  gaps  in  the  development  of  systems  engineers.  It 
was  observed  that  teamwork,  in  particular,  is  a  domain  entry  level  systems  engineers  can 
feel  generally  less  competent  in.  Critical  thinking  and  project  management  had  mixed 
results  of  self-efficacy  levels  between  the  two  cohorts,  with  one  cohort  indicating 
statistically  significant  increases  in  competencies  and  the  other  cohort,  at  best,  indicating 
a  general  sense  of  improvement  (considering  the  difference  in  pre-  and  post-mean 
scores).  Following  Bandura’s  overarching  recommendation  that  “a  major  goal  of  formal 
education  should  be  to  equip  students  with  the  intellectual  tools,  self-beliefs,  and  self- 
regulatory  capabilities  to  educate  themselves  throughout  their  lifetime”  (1993,  136)  the 
research  team  would  advise  NPS  faculty  and  program  developers  to  focus  on  teamwork 
strategies.  For  instance,  teamwork  projects  could  be  adjusted  to  include  upper  and  lower 
classmen.  Research  has  found  that  when  teams  are  structured  to  include  both  upper  and 
lower  classmen,  the  lower  classmen  may  develop  a  heightened  sense  of  competency  in 
regards  to  completing  the  education  because  they  observe  the  upper  classmen  as  models. 
This  instills  in  them  a  “perseverant  self-efficacy”  (Ponton  et  al.  2001,  250). 
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Given  the  research  findings,  the  NPS  team  further  recommends  survey  methods  to 
assess  self-efficacy.  As  stated  earlier,  survey  methods  utilizing  task  specific  measures  in 
the  form  of  a  survey,  might  be  superior  to  other  approaches,  such  as  ER,  to  self-efficacy 
assessment.  Similarly,  the  NPS  research  team  contends  that  alongside  utilizing  the 
COMPOSE  model  as  a  career  development  tool,  the  SE  Competency  Survey  can  be  used 
to  identify  achievement  of  specific  knowledge,  skills  and  abilities  as  indicators  of 
competency  in  performing  systems  engineering  tasks. 

Research  on  the  identification  of  professional  development  KSA  has  suggested 
that  industry  knowledge,  alongside  El,  should  be  included  as  skill  areas  (Moore  and  Rudd 
2005,  69).  Given  the  fact  that  the  NPS  model  does  not  include  industry  knowledge  or  El 
as  competency  areas,  a  recommendation  for  future  research  would  be  to  study  the  impact 
and  feasibility  of  including  the  two  constructs  into  the  NPS  model. 

C.  AREAS  FOR  FURTHER  RESEARCH 

The  NPS’s  research  team  contends  that  the  top  recommendation  for  an  area  for 
further  research  would  be  to  have  an  engineering  organization  validate  the  SE 
Competency  Survey  instrument.  The  NPS  research  team  contends  that  if  possible  within 
time,  participant,  and  resource  constraints,  the  SE  Competency  Survey  can  be  used  by 
other  engineering  disciplines  as  a  model  to  implement  the  self-efficacy  based  method.  It 
can  also  be  tailored  to  meet  an  organization’s  requirements  with  minimal  effort/changes. 
In  referencing  Whitcomb,  Khan  and  White  (2013)  and  White’s  (2013)  research  discussed 
in  Chapter  III,  it  should  be  noted  that  the  KSAs  identified  in  the  COMPOSE  model  have 
been  validated  by  a  couple  of  notable  engineering  organizations.  As  such,  the  validation 
of  the  SE  Competency  Survey  would  be  a  natural  extension  of  the  self-efficacy  research. 
Doing  so  would  provide  necessary  insight  for  those  in  academia  involved  in  the 
education,  training  and  development  of  systems  engineers,  as  well  as  for  those  involved 
in  the  hiring  of  systems  engineers.  To  address  the  concern  of  Moore  and  Rudd  (2005,  75) 
cited  earlier  in  the  thesis,  that  students  in  a  study  such  as  this  are  not  likely  to  perceive 
themselves  to  be  poor  in  a  competency  area  they  think  is  very  important,  another  area  of 
research  could  be  to  develop  a  subjective  SE  exam  for  students  to  take  upon  graduation. 
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Or,  students  could  be  asked  to  relate  the  perceived  level  of  importance  of  a  competency 
alongside  how  competent  they  feel  in  being  able  to  perfonn  it,  for  a  correlation  analysis. 
Similarly,  the  NPS  research  team  recommends  further  research  to  include  a  correlation 
analysis  between  achievement  within  a  particular  SE  domain  and  self-efficacy,  as 
research  indicates  that  a  “high  sense  of  efficacy  in  a  particular  domain  may  not 
necessarily  translate  into  having  [a]  similar  level  of  efficacy  in  another  domain”  (Loo  and 
Choy  2013,  86). 

Finally,  recalling  the  discussion  on  “high  component  loadings”  and  “domain 
specific  KSAs”  the  SE  Competency  Survey  can  be  further  refined  and  the  new  survey 
sub-portions  can  be  re-written  to  include  KSAs  that  are  representative  of  domains  not 
currently  addressed  by  the  survey.  For  instance,  the  SE  Competency  Survey  does  not 
address  El  skill  areas  such  as  time  management  and  balancing  personal  and  professional 
lives,  so  perhaps  this  can  be  an  area  for  further  research.  Keeping  in  mind  that  “in  such 
difficult  budget  times,  demands  on  faculty  time  continue  to  increase”  (Moore  and  Rudd 
2005,  75),  including  El  competencies  may  prove  beneficial  in  providing  infonnation  on 
professional  skills  development  given  the  current  economic  climate. 
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